Arnon Rotem-gal-oz

Using a Mac? Here's an alternative to Docker Desktop.

This post originally published in the AppsFlyer blog. A couple of weeks ago Nir Rubinshtein and I presented AppsFlyer’s data architecture in a meetup ofBig Data & Data Science Israel. One of the concepts that I presented there, which is worth expanding upon is “Data’s Hierarchy of Needs:” Data should Exist Data should be Accessible Data should be Usable Data should be Distilled Data should be Presented How can we make data “achieve its pinnacle of existence” and be acted upon? In other words, what are the areas that should be addressed when designing a data architecture if you want it to be complete and enable creating insights and value from the data you generate and collect. If done properly, your users might just act upon the data you provide. This list might seem a little simplistic but it is not a prescription of what to do but rather a set of reminders of areas we need to cover and questions we need answered to properly create a data architecture. Data Should Exist Well, of course data should exist, and it probably does. You should ask yourself however, is if the data that exists is the right data? Does the retention policy you have service the business needs? Does the availability fit your needs? Do you have all the needed links (foreign keys) to other data so you’d be able to connect it later for analysis? To make this more concrete, consider the following example: AppsFlyer accepts several types of events (launches, in-app events, etc.) which are tied to apps. Apps are connected to accounts (an account would have one or more applications, usually at least, an iOS app and an Android one). If we would save the accounts as the latest snapshot and an app changes ownership, the historical data before that change would be skewed. If we treat the accounts as a slowly changing dimension of the events, then we’d be able to handle the transition correctly. Note that we may still choose to provide the new owner the historic data but now it not the only option the system support and the decision can be based on the business needs. Data Should Be Accessible If data is written to disk it is accessible programmatically at least, however, there can be many levels of accessibility and we need to think about our end users needs and the level of access they’d require. At AppsFlyer, the data existence (mentioned above) is handled by processing all the messages that go through our queues using Kafka but that data is saved in sequence files and stored by event time. Most of our usage scenarios do have a time component but they are primarily handled by the app or account. Any processing that needs a specific account and would access the raw events would have to sift through tons of records (3.7+ billion a day at the time of this post) to find the few relevant ones. Thus, one basic move toward accessibility of data is to sort by apps so that queries will only need to access a small subset of the data and thus run much faster. Then we need to consider the “hotness” of the data i.e. what response times we need and for which types of data. For instance, aggregations, such as retention reports need to be accessed online (so called “sub-second” response), latest counts need near real-time , explorations of data for new patterns can take hours etc. To enable support of these varied usage scenarios, we need to create multiple projections of our data, most likely using several different technologies. AppsFlyer stores raw data in sequence files, processed data in parquet files (accessible via Apache Spark), aggregations and recent data in columnar RDBMS and near real-time is stored in-memory. The three different storage mechanisms I mentioned above (Parquet, columnar RDBMS and In-Memory Data Grid) used in AppsFlyer all have SQL access; this is not by chance. While we (the industry) went through a short period of NoSQL, SQL or almost-SQL is getting back to be the norm, even for semi-structured and poly-structured data. Providing an SQL interface to your data is another important aspect of data accessibility as it allows expanding the user base for the data beyond R&D. Again, this is important not just for your relational data… Data Should Be Usable What’s the difference between accessible data and usable data? For one there’s data cleansing. This is a no-brainer if you pull data from disparate systems but it is also needed if your source is a single system. Data cleansing is what traditional ETL is all about and the techniques still apply. Another aspect of making data usable is enriching it or connecting it to additional data. Enriching can happen from internal sources like linking CRM data to the account info. This can also be facilitated by external sources as with getting the app category from the app store or getting device screen size from a device database. Last but not least, is to consider legal and privacy aspects of the data. Before allowing access to the data you may need to mask sensitive information or remove privacy-related data (sometimes you shouldn’t even save it in the first place). At AppsFlyer we take this issue very seriously and make major efforts to comply when working with partners and clients to make sure privacy-related data is handled correctly. In fact, we are also undergoing independent SOC auditing to make sure we are compliant with the highest standards. To summarize, to make the data usable you have to make sure it is correct, connect it to other data and you need to make sure that it is compliant with legal and privacy issues. Data Should Be Distilled Distilling insights is the reason we perform all the previous steps. Data in itself is of little use if it doesn’t help us make better decisions. There are multiple types of insights you can generate here beginning from the more traditional BI scenarios of slice and dice analytics going through real-time aggregations and trend analysis, ending in applying machine learning or “advanced analytics”. You can see one example of the type of insights that can be gleaned from our data by looking at theGaming Advertising Performance Index we recently published. Data Should Be Presented This point ties in nicely with the Gaming Advertising Performance Index example provided above. Getting insights is an important step, but if you fail to present them in a coherent and cohesive manner then the actual value users would be able to make of it is limited at best. Note that even if you use insights for making decisions (e.g. recommending a product to a user) you’d still need to present how well this decision is doing. There are many issues that need to be dealt with from UX perspective both in how users interact with the data and how the data is presented. An example of the former is deciding on chart types for the data. A simple example for the latter is when presenting projected or inaccurate data it should be clear to the users that they are looking at approximations to prevent support calls on numbers not adding up. Making sure all the areas discussed above are covered and handled properly is a lot of work but providing a solution that actually helps your users make better decisions is well worth it. The data’s hierarchy of needs is not a prescription of how to get there, it is merely a set of waypoints to help navigate toward this end goal. It helps me think holistically about AppsFlyer data needs and I hope following this post it would also help you. For more information about our architecture, check out the presentation from the meetup: Architecture for Real-Time and Batch Big Data Analytics Distilling insights @ AppsFlyer

It has been few months since SOA Patterns was published and so far the book sold somewhere between 2K-3K copies which I guess is not bad for an unknown author – so first off, thanks to all of you who bought a copy (by the way, if you found the book useful I’d be grateful if you could also rate it on Amazon so that others would know about it too) I know at least a few of you actually read the book as from time to time I get questions about it :). Not all the questions are interesting to “the general public” but some are. One interesting question I got is about the so called “Canonical schema pattern“. I have a post in the making (for too long now,sorry about that Bill) that explains why I don’t consider it a pattern and why I think it verges on being an anti-pattern. Another question I got more recently, which is also the subject of this post, was about the Saga pattern. Here is (most of) the email I got from Ashic : “Garcia-Molina’s paper focuses on failure management and compensation so as to prevent partial success. It discusses a variety of approaches – with an SEC, with application code outside of the database, backward-forward and even forward-only (the latter having no “compensate” step per activity, rather a forward flow that takes care of the partial success). Nowadays, I see two viewpoints regarding sagas: 1. People calling process managers sagas, which is obviously incorrect. [e.g. NServiceBus "sagas".] 2. People focusing very strongly on a “context” of work, whereby the context gets passed around from activity to activity. For linear up front workflows, routing slips are an easy solution. An example of this can be found at Clemens’s post here: http://vasters.com/clemensv/2012/09/01/Sagas.aspx . For more complicated workflows, graph-like slips may be used. After discussing with some enthusiasts, they seem very keen to suggest that the context has to move along. They seem to reject the notion of a saga where a central coordinator controls the process. In other words, even if a process manager takes care of only routing messages, and that routing includes compensations to alleviate partial successes, they are unwilling (sometimes vehemently) to call that a saga. They acknowledge it can be useful, but say that is not a saga. I find this to be confusing. In this case the process manager acts as the SEC would in a Garcia-Molina saga capable database. This approach still allows interleaved transactions (or steps) without a global lock. Why would this not be a saga? In your book, I did see you mentioned orchestration as a way of implementing sagas. However, when this was brought up, the proponents of point 2 suggest that that is not what you really mean. To me it seems quite clear, and it aligns with Hector’s paper. I just want to make sure I have this right. I’d love your thoughts on this.” Let’s start with the answer to the question: When I think about the Saga pattern I see it as the application of the notions in the Garcia-Molina paper (which talked about databases) to SOA. In other words, I see sagas as the notion of getting distributed agreement of a process with reduced guarantees (vs. distributed transactions that propose ACID guarantees across systems). – So,basically, a Saga is loose transaction-like flow where, in case of failures, involved services perform compensation steps (which may be nothing, a complete undo or something else entirely). The Saga pattern can augment this process with temporary promises (which I call reservations). Under this definition both centrally managed processes and a “choreographed” processes are Sagas – as long as the semantics and intent mentioned above are kept. The centrally managed orchestration provides visibility of processes, ease of management etc; The cooperative event based, context shared sagas provide flexibility and allow serendipity of new processes; Both have merit and both have a place, at least in my opinion :) The main reason both of these, very different, approaches are valid designs and implementations for the Saga pattern is that the Saga pattern (like others in the book) is an Architectural pattern and not a Design pattern. Which brings us to the second reason for this post, the difference between “Architecture” and “Design”. In a nutshell, architecture is a type of design where the focus is quality attributes and wide(er) scope whereas design focuses on functional requirements and more localized concerns. The Saga pattern is an architectural pattern that focused on the integrity reliability quality attributes and it pertains to the communication patterns between services. When it comes to design the implementation of the pattern. you need to decide how to implement the concerns and roles defined in the pattern -e.g. controlling the flow and the status of the saga. One decision can be to implement it centrally and use orchestration another decision can be to decentralize it and use context… Design decision can be very meaningful sometimes it can be hard to find what’s left of the architecture – consider for example the whole idea behind blogging and RSS feeds. The architectural notion is a publish/subscribe system where the blog writer publish an “event” (a new post) and subscribers get a copy. When it came to design and implementation, considering it was implemented on top of HTTP and REST where there is no publish/subscribe capability it was actually designed as a pull system where the publisher provides a list of recent changes (the feed) and subscribers sample it and check if anything changed since the last time. So architecturally pub/sub, design pull a centralized server that exposes latest changes – a really big difference Does it matter at all? I think yes. Architecture lets us think about the system at a higher level of abstraction and thus tackle more complex systems. When we design and focus on more local issues we can tackle the nitty gritty details and make sure things actually work. we need to check the effects of design on architecture and vice versa to make sure the whole thing sticks together and actually does what we want/need. Note that architecture and design are not the complete story – another variable is the technology (e.g. HTTP in the example above) which affects the design decision and thus also the architecture (you can read a little more about it in my posts on SAF)

i recently answered this question in stackoverflow : what is an utility tree and what is it’s purpose in case of architecture tradeoff analysis method(atam)? i did answer the question there but here’s a better explanation with lots of examples based on the initial version for chapter 1 of soa patterns (which didn’t make it into the final version of the book). there are two types of requirements for software projects: functional and non-functional requirements. functional requirements are the requirements for what the solution must do (which are usually expressed as use cases or stories). the functional requirements are what the users (or systems) that interact with the system do with the system (fill in an order, update customer details, authorize a loan etc.). non-functional requirements are attributes the system is expected to have or manifest. these usually include requirements in areas such as performance, security, availability etc. a better name for non-functional requirements is “quality attributes” . below are some formal definitions from ieee standad 1061 “standard for a software quality metrics methodology” for quality attributes and related terms: quality attribute a characteristic of software, or a generic term applying to quality factors, quality subfactors, or metric values. quality factor a management-oriented attribute of software that contributes to its quality. quality subfactor a decomposition of a quality factor or quality subfactor to its technical components. metric value a metric output or an element that is from the range of a metric. software quality metric a function whose inputs are software data and whose output is a single numerical value that can beinterpreted as the degree to which software possesses a given attribute that affects its quality. most of the requirements that drive the design of a software architecture comes from system’s quality attributes. the reason for this is that that the effect of quality attributes is usually system-wide (e.g. you wouldn’t want your system to have good performance only in the ui – you want the system to perform well no matter what) – which is exactly what software architecture is concerned with. note however, that few requirements might still come from functional requirements) [1] . the question is how do we find out what those requirements are? the answer to that is also in the software architecture definition. the source for quality attributes are the stakeholders. so what or who are these “stakeholders”? well, a stakeholder is just about anyone who has a vested interest in the project. a typical system has a lot of stakeholders starting from the (obvious) customer, the end-users (those people in the customer organization/dept that will actually use the software) and going to the operations personnel (it – those who will have to keep the solution running), the development team, testers, maintainers, management. in some systems the stakeholders can even be the shareholders or even the general public (imagine for example, that you build a new dispatch system for a 911 center). one of the architect’s roles is to analyze the quality attributes and define an architecture that will enable delivering all the functional requirements while supporting the quality attributes. as can be expected ,sometimes quality attributes are in conflict with each other – the most obvious examples are performance vs. security or flexibility vs. simplicity and the architect’s role is to strike a balance between the different quality attributes (and the stakeholders) to make sure the overall quality of the system is maximized. contextual solutions (e.g. patterns) can be devised to solve specific quality attributes need. however saying that a system needs to have “good performance” or that it needs to be “testable” doesn’t really help us know what to do. in order for us to be able to discern which patterns apply to specific quality attribute , we need a better understanding of quality attributes besides the formal definition, something that is more concrete. the way to get that concrete understanding of the effect of quality attributes is to use scenarios. scenarios are short, “user story”-like proses that demonstrate how a quality attribute is manifested in the system using a functional situation quality attributes scenarios originated as a way to evaluate software architecture. the software engineering institute developed several evaluation methodologies, like architecture tradeoff analysis method (clements, kazman and klein, 2002) that heavily build on scenarios to contrast and compare how the different quality attributes are met by candidate architectures. atam (and similar evaluation methods like laaam which is part of msf 4.0) suggest building a “utility tree” which represent the overall usefulness of the system. the scenarios serve as the leafs of the utility tree and the architecture is evaluated by considering how the architecture makes the scenarios possible. i found that using scenarios and the utility tree approach early in the design of the architecture (see writings about saf ) can greatly enhance the quality of the architecture that is produced. when you examine the scenarios you can also prioritize them and better balance conflicting attributes. the scenarios can be used as an input to make sure the quality attributes are actually met. furthermore you can use the scenarios to help identify the strategies or patterns applicable to make the scenarios possible (and thus ensure the quality attributes are met) within the system. we usually group scenarios into a “utility tree” which is a representation of the total usefulness (“utility”) of a system . as you can see in the diagram below we have the key quality attributes (performance, security etc.). each of the quality attributes has sub categories (e.g. performance is broken into latency, data loss etc.). each sub category is demonstrated by a scenario that we expect the system to manifest. the tree representation helps get the whole picture but the important bits here are the scenarios so let’s explore them some more. scenarios are expressed as statements that have 3 parts: a stimulus , a context and a response . the stimulus is the action taken (by the system / user/ other system / any other person); response is how the system is expected to behave when the stimulus occur, and the context specifies the environment or conditions under which we expect the to get the response. for example in the following scenario: “when you perform a database operation , under normal condition, it should take less than 100 miliseconds.” “under normal condition” is the context “when you perform a database operation” is the stimulus “it should take less than 100 millisecond” is the response expected from the system. here are a couple of additional examples for quality attribute scenarios: performance –>latency -> under normal conditions a client consuming multiple services should have latency less than 5 seconds. security->authentications -> under all conditions, any call to a service should be authenticated using x.509 certificate you can also check out this document for a few more scenario examples from a system i worked on in the past [1] design has the ratios reversed i.e. most of the requirements for design come from functional requirements and a few requirements might come from the quality attributes. illustration by epsos.de

After a long hiatus, I guess it is time for another SOA anti-pattern to see the light. It is probably also a good time to remind you that I am looking for your insights on this project. In any event I hope you’d find this anti-pattern useful and as always comments are more than welcomed (do keep in mind this is an unedited draft :) ) ------------------------------------- There are many unsolved mysteries, you’ve probably heard about some of them like the Loch Ness monster, Bigfoot etc. However, the greatest mystery, or so I’ve heard, is getting the granularity of services right… Kidding aside, getting right-sized services is indeed one of the toughest tasks designing services – there’s a lot to balance here e.g. the communications overhead, the flexibility of the system, reuse potential etc. I don’t have the service granularity codex and deciding the best granularity depends on the specific context and decisions (e.g. the examples in the Knot anti-pattern above). It is an easier task to define what shouldn’t be a service for instance, calling all of your existing ERP system a single service should definitely be shunned. The Nanoservices anti-pattern talks about the other extreme… the smaller services Consider, for instance, the “calculator service” which appears in samples web-wide (I’ve personally seen examples in .NET, Java, PHP, C++ and a few more). A basic desk calculator, as we all know, supports several simple operations like add, subtract, multiply and divide and sometimes a few more. Implementing a calculator service isn’t very complicated - Listing 10.1 below, for example, shows part of WSDL for a java calculator service that, lo and behold, accepts two numbers and adds them. Listing 10.1 excerpt from a WSDL of a stateless calculator service example. The sample only includes the data needed for the “Add” operation. The add operation accepts two numbers and returns a result (http://cwiki.apache.org/GMOxDOC21/jaxws-calculator-simple-web-service-with-jax-ws.html) Calculator services can be even more advanced and have memory - consider listing 10.2 below, which shows an interface definition for a .NET (WCF) sample that uses workflow services and accepts a single value at a time Listing 10.2 a Service contract definition for a statufil calculator service (http://msdn.microsoft.com/en-us/library/bb410782.aspx). The service accepts a single number at a time and remembers the former state from operation to operation. [ServiceContract(Namespace = "http://Microsoft.WorkflowServices.Samples")] public interface ICalculator { [OperationContract()] int PowerOn(); [OperationContract()] int Add(int value); [OperationContract()] int Subtract(int value); [OperationContract()] int Multiply(int value); [OperationContract()] int Divide(int value); [OperationContract()] void PowerOff(); } The calculator service (both versions of it) is a very fine grained service. Naturally, or hopefully anyway, the calculator examples are just over simplified services used to demonstrate SOA related technologies (JAX-WS in the first excerpt and WCF and WF in the second one). The problem is when we see this level of granularity in real life services 1.1.1Consequences Problem? Why is “fine granularity” a problem anyway? Isn’t SOA all about breaking down monolith “silos” into small reusable services? More so, the finer grained a service is, the less context it carries. The less context a service carries the more reuse potential it has – and reuse is one of the holy grails of SOA isn’t it? The calculator service above seems like the epitome of a reusable service. There’s no doubt we can reuse it over and over and over. Reuse is indeed a noble goal (I’ll leave discussing how real it is for another occasion), the culprit of fine grained services, however, is the network. Services are consumed over networks – both local (LANs) and remote (extranets, WANs etc.). The result is that services are bound by the limitations and costs incurred by those network. Trying to disregard these costs is exactly what ailed most, if not all, RPC distributed system approaches that predated SOA (Corba, DCOM etc.) - The calculator service and other similarly sized services are nanoserivces. Nonoservice is an Anti-pattern where a service is too fine grained. Nanoservice is a service whose overhead (communications, maintenance etc.) out-weights its utility. So how can nanoservices harm your SOA? Nanoservices cause many problems, the major ones being poor performance, fragmented logic and overhead. Let’s look at them one by one Every time we send a request to a service we incur a few costs such as serialization on caller, moving caller process to the OS network service, translation to the underlying network protocol, traveling on the network, moving from the OS network service to the called process, deserialization on the called process – and that’s before adding security (encryption, firewalls etc), routing , retries etc. Modern networks and servers can make all this happen rather fast but if we have a lot of nano-services running around these numbers add-up to a significant performance nightmare, Nano-services cause fragmented logic - almost by definition. As we break what should have been a meaningful cohesive service, into miniscule steps our logic is scattered between the bits that are needed to complete the business service. The fact that you need to haul over several services to accomplish something meaningful also spell increased chances of the Knot anti-pattern, mentioned above. Proliferation of Nanoservices also causes development and management overhead. Just look at the amount of WSDL needed to define the calculator services in listing 10.1 above and for what? A service that adds a couple of numbers… There is a relatively fixed overhead associated with managing a service. This include things like keeping track of a service in a service registry, making sure it adheres to policy, writing the cruft (things we have to write around the business logic) for configuring it etc. Having nano-services around means we have to do this a whole-lot more times (i.e. per service) compared with having fewer coarser grained services. The point of overhead out-weighing utility that appears in the Nano-services definition above is subtle but important. The fact that a contract does not have a lot of operations means we want to make sure we don't have a nano-service, but it doesn't automatically mean that it is. For instance, a fraud detection service contract might only accept transaction details and decide whether to authorize the transaction, deny it or move to further investigation. However the innards of this service involve a complex process like running the details in a rule engine checking for fraudulent behavior patterns, matching to black lists etc. In fact Fraud detection is such a complicated issue that these are actually systems and a SOA based one would be comprised of several services in itself. The other side of the equation is also true a comprehensive contract does not guarantee a service is not a nano-service. For instance, in a system I designed on the initial iterations we developed a resource management service. It supported some very nice operations like getting status of all the services in the system, running sagas and of course allocating services. Allocating services meant that whenever an event went out that needed a (new) service instance to handle it, we had to make a call to the resource manager to get one. This provides for a neat centralized management and also for a performance bottleneck that slows the whole system. To solve this we went with distributed resource management but that't beyond the scope of this discussion. The point, however that is that the utility of the resource management (e.g. easy management of running sagas ) vs. overhead associated with the service (the number of calls and performance hit on the system) was not worth it – Hench a nano-service. 1.1.2Causes From a more technical point of view, we get to nanoservices from not paying attention to at least a couple of the fallacies of distributed computing. Mentioned in chapter 1, the fallacies of distributed computing are a few false assumptions that are easy to make and prove to be wrong and costly down the road. Specifically, we are talking here about assuming that § Bandwidth is infinite – Even though bandwidth gets better and better, it is still not infinite within a specific setup. For instance in one project we were sending images over the wire and distribute them to computational services (a la map/reduce – see also Gridable Service in chapter 3). Things were working ok when we sent small images, but when we sent larger images we understood we were sending them as bitmaps and not as much more compact jpegs which caused a burden on the backbone of our switches which wasn’t ready for that load. § Transport cost is zero – As explained in the previous section every over-the-wire call incurs a lot of costs vs. a local call (also see figure 10.5 below).The costs of the transport can be considered both from the time it take to make each of these calls but even the real dollar value attached to making sure you have enough bandwidth (connection/routers, firewalls) to handle the traffic incurred Figure 10.5 Local objects can “afford” to have intricate interactions with their surroundings. A similar functionality delivered over a network is more likely than not to cause poor performance because of the network related overhead. Another reason to get Nano-Services, at least for beginners are poor examples – as, noted the calculator services above are taken from real examples provided by various vendors. SOA newcomers and/or people without a lot of distributed systems development experience can be easily take these samples at face value, and go about implementing services with similar granularity. The fact that that web-service framework mostly map service calls to object method calls makes this even more tempting. Nano-services is also an inherent risk when applying the orchestrated choreography pattern. Adding an orchestration engine, capable of controlling flow and external to services tempts us to think that we can use it to drive all flow as little as it may seem. Couple this with the fact that the smaller the services are the more “Reuseable” they are (less context) and, again, you may end up with a lot of nano-services on your hands. Lastly, since the nano-services boundary is soft (remember utility vs. overhead weight) behaviors that can look promising at design time can prove to be nano-services moving along (like the resource manager example above). This can be an acceptable if your SOA is developed iteratively (see 10.2.4 exceptions below) but it still mean that we have to come up with ways to refactor nano-services. 1.1.3Refactoring There are basically two main ways to solve the nano-services problem. One, which is relatively easy, is to group related nano-services into a larger service. The second option, which is more complicated, is to redistribute its functionality among other services. Let's take a look at them one by one. On one project I was working on we needed to send out notifications to users and admins via SMS messages. Since the software component that did the actual SMS dissemination was a 3rd party app we’ve decided to create a simple service (not unlike OO adapter) that accepts requests for SMS and talks to the 3rd party software. A nano-service was born, it even got a nice little name Post Office Service (ok, ok the original name was Spam Server but I thought it would look bad in presentations J). Why is this a nano service? Well, it really doesn’t do much and it would be even simpler to package this as a library that other services can use and it does have all the management overhead of maintain as another system service. What we did about it was to add similar functionality to the service so it also learned to send emails, tweets and MMSs. A serendipitous effect of this was that now instead of sending a request like TweetMessage or SendSMS to this service we could now raise more meaningful events such as SystemFailureEvent and have the service make decisions on how to alert administrators based on the severity of the problem etc. So combining the related functionality helped make the overall service even more meaningful. Unfortunately it isn’t always possible to take the functionality of Nano-services and find suitable “other services” (nano-or right-sized) that can assimilate them. In those cases getting rid of a nano-service is more of an exercise in redesign than is a refactoring. For instance, in a project we’ve built we had a services allocation service (SAS). The SAS role was to know about other services location and health status and utilization and upon a request, such as beginning a Saga (see chapter 5 for the saga pattern) decide what service instances should be used. The service also provided “reporting” capabilities for active sagas, services utilization etc. This might not sound like a nano-service, and at first we thought so too, but as the project progressed we found that being a central hub, as seen in figure 10.6 below, made the SAS a performance bottleneck, incurring additional costs (in latency) on a lot of the calls and interactions made by other services. The utility of the SAS, of finding what service instance to talk to, was being diminished by the cost – yep it is a nano-service after all. Figure 10.6 An example for a nano-service. The SAS service is a performance bottleneck as a lot of calls go through it. It provides an important service but the costs of its are too dear. To solve the SAS problem we had to put in quite a lot of work. The solution, essentially was to move to distributed resource management, so that each service had some knowledge of what the world looks like so that it could decide what service instant to talk to by itself. To sum this section, sometimes it is easy to notice that something is a nano-services, chances are that in these cases it would also be easy to take the functionality and group it with related functionality in other services. However on other occasions the fact that a service provides too little benefit is not as apparent and only becomes clear as we move along. In those cases it is also harder to fix the problem. One question we still need to cover is are there any situations where we would go with a nano-service even if we know it is one on the onset. 1.1.4Known Exceptions When is it ok to have Nano-services? When you are starting out. When your approach to SOA is evolutionary and you don’t plan everything in advance (something that rarely work anyway, but that’s another story), there’s a good chance that first versions of services you build will not show a lot of business benefit, but they will already need the full overhead of a service. The post office service in the example above is a good example for that as starting out it only dealt with a single type of message and it didn’t do a whole lot with it either. The post office service is also a good example for another reason to have a nano-service which is when you want to build an adapter or bridge to other systems be that legacy systems or 3rd party ones. In these cases you need to weight the advantage of using a service vs. building the same functionality as a library that can be used within services, but in many cases keeping the flexibility and composability of SOA can triumph over the overhead associated with having an additional service to manage. Lastly, one point to keep in mind is that NanoServices is a rather soft pattern and the value of a small service can radically change from system to system or even in a certain system as time and requirements progress. It is worthwhile questioning our assumptions and looking at the services that we grow from time to time to validate the usefulness of what we’re building.

Earlier today I read a post by Michael Feathers Called "10 Papers Every Developer Should Read (At Least Twice). I knew some of the articles mentioned there and learnt about few interesting ones.I liked it so much, I thought I'd compile a similar list for software architects - based on stuff I read over the years. The Byzantine Generals Problem (1982) by Leslie Lamport, Robert Shostak and Marshall Pease - The problem with distributed consensus Go To statements considered harmfull (1968) - by Edsger W. Dijkstra - Didn't you always want to know why ? :) A Note on Distributed Computing (1994) - by Samuel C. Kendall, Jim Waldo, Ann Wollrath and Geoff Wyant - Also on Michael's list but it is one of the foundation papers on distributed computing Big Ball of Mud (1999) - Brian Foote and Joseph Yoder - patterns or anti-patterns? No Silver Bullet Essence and Accidents of Software Engineering (1987) - Frederick P. Brooks - On the limitations of Technology and Technological innovations. The Open Closed Principle (1996) - Robert C. Martin (Uncle Bob) - The first in a series of articles on Object Oriented Principles (you remember the debate on SOLID...) IEEE1471-2000 A recommended practice for architectural description of software intensive systems (2000) various- It is a standard and not a paper but it is the best foundation for describing a software architecture I know. Harvest, Yield, and Scalable Tolerant Systems (1999) Armando Fox, Eric A. Brewer - That's where the CAP theorem was first defined An Introduction to Software Architecture (1993) - David Garlan and Mary Shaw - one of the foundation articles of software architecture field (although based on earlier work by the two) Who Needs an Architect? (2003) Martin Fowler - Do we or don't we? I could come up with quite a few more articles not to mention books that aren't in this list. However these are definitely some of the most influential papers I read

In a post called "Rhino Service Bus: Saga and State" Ayende said "In a messaging system, a saga orchestrate a set of messages. The main benefit of using a saga is that it allows us to manage the interaction in a stateful manner (easy to think and reason about) while actually working in a distributed and asynchronous environment." I really don't agree with this definition of a saga. The Saga provides a context for set of messages to allow manging an effort for distributed concensus. It does not "orchestrate" messages (that's what workflows are for) - you can read more on Saga's in an excerpt from my SOA patterns book: Saga pattern. Here's the comment I left on Ayende's site: "What you describe is nice except it isn't a Saga it is more of a workflow. The notion of Saga which is originated from databases relates to the overall coordination of state between the different services - or the context for the whole business process. In the coffee shop example you use that would be the whole "transaction" from the point the customer orders her coffee until she either gets it or the transaction is canceled (e.g. it took too long and the customer leaves or the coffee shop is out of milk etc.) Unlike database (or distributed) transaction when/if a saga is aborted the different component of the system might not return to their previous state e.g. if the customer complains that the coffee is not good and gets her money back. the milk is not separated back from the coffee beans and returned to the bottle - rather the coffee cup goes to the trash. Workflow is one strategy a service can take to handle the long running interaction within a saga. In your case the BristaSaga class (which I think should be BristaWF) orchestrate the internal state transitions depending on the different messages that arrive within the saga. In your case you have a hardcoded workflow - but it is also possible to use a workflow engine for the job. By the way, in the above example you could also use a statemachine instead of a WF to manage the process " In another comment Kristofer asked me: Arnon: I'm not 100% sure of how you distinguish a Saga from a Workflow, could you elaborate some more on this? A Saga involves a number of underlying workflows? A Saga might as well contain a number of underlying Sagas? Isn't it just a question of at what level it is initiated? If a Saga should represent the whole transaction / business process, then who should handle it? Couldn't it be implemented as a Saga, exactly as Ayende describes it, by the initiating service (in this case the ordering)?, which then also is given the responsibility to handle restoring the total state etc of underlying/involved services if the transaction is aborted? The possibility to restore state does of course depend on what the specific Saga is handling, some processes might not be able to "rollback" completely, it's rather a question of rolling back all involved parties to a known/acceptable state." The answer is that ,again, Saga is similar to a transaction in the sense that it provides a shared context for an attempt to get a distributed consensus Unlike a transaction which insures ACID properties. Sagas are not. The concept of dissipating that shared context, having each party (service) affect whether the saga should be aborted or successful etc. is what I call a saga. When a saga is aborted the only thing the coordinator can do is pass the status to the participants. Each of the services is responsible to do its best effort to handle the abort (either by rolling back, compensation or whatever) Workflow is another thing altogether. which keeps a context between calls and means externalizing the decisions on the logic flow from the business logic (usually with a workflow engine). You can use workflows within a service (a pattern I call workflodize) or you can use them externally (a pattern I call orchestrated choreography e.g. BPM) You can use either form of workflow to support the implementation of a saga but you can also implement sagas without workflows. In our system we use an "event broker" (see www.rgoarchitects.com/.../EventingInWCF.aspx) the event broker infrastructure dissipates the saga context when you raise a saga event. A service that initialized a saga (by sending the first event) can choose to close the saga (commit) or abort it. etc. We don't currently have any workflow driven services (but some of them use a state machine as an alternative) (I think the term Saga does not describe Ayende's class since the "barista" is just on of the participants in the saga there are other participants.)