Written by Leena Joshi Microsoft recently announced that Microsoft AppFabric 1.1 for Windows Server will be at the end of support on April 2, 2016. Less than a year away! Don’t panic yet, there is another, better solution. Redis is the product of choice for thousands of developers worldwide who want to accelerate their applications. Redis is the fastest growing NoSQL datastore, that runs in-memory and delivers millions of transactions at sub millisecond latencies. Redis Labs provides enterprise class Redis: as a downloadable product Redis Labs Enterprise Cluster (RLEC) or as a seamlessly scalable, highly available service, Redis Cloud. Here are the top 5 reasons to move applications using Microsoft AppFabric to Redis Cloud or Redis Labs Enterprise Cluster : Performance: One of the biggest reasons to move is the blazing fast performance and versatility of Redis. While AppFabric performs simple GET and SET commands, Redis comes with a variety of data structures (strings, lists, hashes, sets, sorted sets) and a sophisticated set of commands, embedded Lua scripting and bit operations that helps you address more than high speed caching – it lets you implement high speed transactions, real time analytics, in-app social functionality, messaging, job and queue management and much more. Scalability: The current AppFabric replacement offered by MS runs on Azure, however it is not a service that scales seamlessly like Redis Cloud. Redis Cloud with 4900+ customers to date is proven to be highly scalable and available and provides all the functionality of Redis with none of the deployment overhead. Also, if you don’t really want a cloud solution, RLEC (Redis Labs Enterprise Cluster) runs on-premises or wherever you are deployed and relieves you of any provisioning, configuring, scaling, clustering, monitoring – it fully automates all those tasks and makes it super-easy to deploy Redis clusters. Portability: AppFabric is Windows and .Net specific while Redis supports many environments and languages.Thanks to our community, Redis supports many languages including Python, Ruby, Java, PHP, Node, C, C# . So, if you have a mixed environment, now you can even extend your use of in-memory, high speed technologies. Built-in Monitoring: Both Redis Cloud and RLEC come with a dashboard to view different operational metrics as well as built-in alerts to receive notification on important events. This is very much more cumbersome with AppFabric and you would have had to use external tools to get the same level of manageability. Ease of use: Redis is simple yet very sophisticated and used by thousands of developers worldwide. It is #3 among NoSQL database adoption (source:DBengines) and #12 among all tools used by developers (source:Stackshare). This means that a talent pool of Redis users and a worldwide community is always there to help! RLEC is free to download and Redis Cloud has a free tier as well – so it is really easy to try out both those products. If you have any questions about migrating from AppFabric to Redis Labs, our solutions consultants are always here to help. Email us at [email protected] and we can set up some time to walk you through the migration! - See more at: https://redislabs.com/blog/appfabric-coming-apart-top-5-reasons-to-move-to-redis#.VYSGzucYGL4

In a previous blog post, I introduced the concept of “query autofiltering”, which is the process of using the meta information (information about information) that has been indexed by a search engine to infer what the user is attempting to find. A lot of the information used to do faceted search can also be used in this way, but by employing this knowledge up front or at “query time”, we can answer questions right away and much more precisely than we could without techniques like this. A word about “precision” here – precision means having fewer “false positives” – unintended responses that creep in to a result set because they share some words with the best answers. Search applications with well tuned relevancy will bring the best results to the top of the result list, but it is common for other responses, which we call “noise hits”, to come back as well. In the previous post, I explained why the search engine will often “do the wrong thing” when multiple terms are used and why this is frustrating to users – they add more information to their query to make it less ambiguous and the responses often do not reward that extra effort – in many cases, the response has more noise hits simply because the query has more words. The solution that I discussed involves adding some semantic awareness to the search process, because how words are used together in phrases is meaningful and we need ways to detect user intent from these patterns. The traditional way to do this is to use Natural Language Processing or NLP to parse the user query. This can work well if the queries are spoken or written as if the person were asking another person, as in “Where can I find restaurants in Cleveland that serve Sushi?” Of course, this scenario –which goes back to the early AI days – has become much more important now that we can talk to our cell phones. For search applications like Google with a “box and a button” paradigm, user queries are usually one word or short phrases like “Sushi Restaurants in Cleveland”. These are often what linguists call “noun phrases” consisting of a word that means a person, place or thing (what of who they want to find or where) – e.g. “restaurant” and “Cleveland” and some words that add precision to their query by constraining the properties of the thing they want to find – in this case “sushi”. In other words, it is clear from this query that the user is not interested in just any restaurant – they want to find those that serve raw fish on a ball of rice or vegetable and seafood thingies wrapped in seaweed. The search engine often does the wrong thing because it doesn’t know how to combine these terms – and typically will use the wrong logical or boolean operator – OR when the users intent should be interpreted as AND. It turns out that in many cases now, our search indexes know the difference between Mexican Restaurants (which typically don’t serve Sushi) and Japanese Restaurants (which usually do) because of the metadata that we put into them to do faceted search (Funny story: after posting this, I ran across a Mexican Restaurant in Toms River, New Jersey that does serve sushi – but still, most of them don’t!). The goal of query autofiltering is to use that built in knowledge to answer the question right away and not wait for the user to “drill in” using the facets. If users don’t give us a precise query (like simply “restaurants”), we can still use faceting, but if they do, it would be cool if we could cut to the chase. As you’ll see, it turns out that we can do this. The previous post contained a solution which I called a “Simple” Category Extraction component. It works by seeing if single tokens in the query matched field values in the search index (using a cool Lucene feature that enable us to mine the “uninverted” index for all of the values that were indexed in a field). For example, if it sees the token “red” and discovers that “red” is one of the values of a “color” field, it would infer that the user was looking for things that are “red” in “color” and will constrain the query this way. The solution works well in a limited set of cases, but there are a number of problems with it that make it less useful in a production setting. It does a nice job in cases where the term “red” is used to qualify or more precisely specify a thing – such as “red sofa”. It does not do so well in cases where the term “red” is not used as a qualifier – such as when it is part of a brand or product name such as “Red Baron Pizza” or “Johnny Walker Red Label” (great Scotch, but “Black Label” is even better, maybe I’ll be rich enough to afford “Blue Label” some day – but I digress …). It is interesting to note that the simple extractor’s main shortcomings are due to the fact that it looks at single tokens at a time in isolation from the tokens around it. This turns out to be the same problem that the core search engine algorithms have – i.e., it’s a “bag of words” approach that doesn’t consider – wait for it – semantic context. The solution is to look for patterns of words that match patterns of content attributes. This does a much better job of disambiguation. We can use the same coding trick as before (upgraded for API changes introduced in Solr 5.0), but we need to account for context and usage – as much as we can without having to introduce full-blown NLP which needs lots of text to crunch. In contrast, this approach can work when we just have structured metadata. Searching vs Navigating A little historical background here. With modern search applications, there are basically two types of user activities that are intermingled: searching and navigating. The former involves typing into a box and the latter, clicking on facet links. In the old days, there was a third user interface called an “advanced” search form where users could pick from a set of metadata fields, put in a value and select their logical combination operators– an interface that would be ideally suited for precise searching given rich metadata. The problem is that nobody wants to use it. Not that people ever liked this interface anyway (except those with Master of Library Science degrees), but Google has also done much to demote this interface to a historical reference. Google still has the same problem of noise hits but they have built a reputation for getting the best results to the top (and usually, they do) – and they also eschew facets (they kinda have them at the bottom of the search page now as related searches). Users can also “markup” their query with quotation marks or boolean expressions or ‘+/-’ signs but trust me – they won’t do that either (typically that is). What this means is that the little search box – love it or hate it – is our main entry point – i.e. we have to deal with it, because that is what users want – to just type stuff and then get the “right” answer back. (If poor ease-of-use or the simple joy of Google didn’t kill the advanced search form completely, the migration to mobile devices absolutely will). A Little Solr/Lucene Technology – String fields, Text fields and “free-text” searching: In Solr, when talking about textual data, these two user activities are normally handled by two different types of index field: string and text. String fields are not analyzed (tokenized) and searching them requires an exact match on a value indexed within a field. This value can be a word or a phrase. In other words, you need to use : syntax in the query (and quoted “value here” syntax if the query is multi-term) – something that power users will be OK with but not something that we can expect of the average user. However, string fields are very good for faceted navigation. Text fields on the other hand are analyzed (tokenized and filtered) and can be searched with “freetext” queries – our little box in other words. The problem here is that tokenization turns a stream of text into a stream of tokens (words) and while we do preserve positional information so we can search on phrases, we don’t know a priori where those phrases are. Text fields can also be faceted (in fact, any field can be a facet field in Solr), but in this case, the facets are based on individual tokens which don’t tend to be too useful. So we have two basic field types for text data, one good for searching and one for navigating. In the harder-to-search type, we know exactly where the phrases are but we typically don’t in the easier-to-search type. A classic trade-off scenario. Since string fields are harder to search (at least within the Google paradigm that users love), we make them searchable by copying their data (using the Solr “copyField” directive) into a catchall text field called “text” by default. This works, but in the process we throw away information about which values are meant to be phrases and which are not. Not only that, we’ve lost the context of what these values represent (the string fields that they came from). So although we’ve made these string fields more searchable, we’ve had to do that by putting them into a “bag of words” blender. But the information is still somewhere in the search index, we just need a way to get it back at at “query time”. Then, we can both have our cake AND eat it! Noun Phrases and the Hierarchy of meta information When we talk about things, there are certain attributes that describe what the thing is (type attributes) and others that describe the properties or characteristics of the thing. In a structured database or search index, both of these kinds of attributes are stored the same way – as field/value pairs. There are however, natural or semantic relationships between these fields that the database or search engine can’t understand, but we do. That is, noun phrases that describe more specific sets of things are buried in the relationships between our metadata fields. All we have to do is dig them out. For example, if I have a database of local businesses, I can have a “what” field like business type that has values like “restaurant”, “hardware store”, “drug store”, “filling station” and so forth. Within some of these business types like restaurant, there may be refining information like restaurant type (“Mexican”, “Chinese”, “Italian”, etc) or brand/franchise (“Exxon”, “Sunoco”, “Hess”, “Rite-Aid”, “CVS”, “Walgreens”, etc.) for gas stations and drug stores. These fields form a natural hierarchy of metadata in which some attributes refine or narrow the set of things that are labeled by broader field types. Rebuilding Context: Identifying field name patterns to find relevant phrase patterns So now its time to put Humpty Dumpty back together again. With Solr/Lucene – it is likely that the information that we need to give precise answers to precise questions is available in the search index. If we can identify sub-phrases within a query that refer or map to a metadata field in the index, we can then add the appropriate metadata mapping on behalf of the user. We are then able to answer questions like “Where is the nearest Tru Value hardware store?” because we can identify the phrase “Tru Value” as a business name and “hardware store” as a specific type of store. Assuming that this information is in the index in the form of metadata fields, parsing the query is a matter of detecting these metadata values and associating them with their source fields. Some additional NLP magic can be used to infer other aspects of the question such as “where is the nearest”, which should trigger the addition of a spatial proximity query filter for example. The Query AutoFiltering Search Component To implement the idea set out above, I developed a Solr Search Component called QueryAutoFilteringComponent. Search components are executed as part of the search request handling process. Besides executing a search, they can also do other things like spell checking or query suggestion, return the set of terms that are indexed in a field or the term vectors (term frequency statistics) among other things. The SearchComponent interface defines a number of methods one of which – prepare( ) – is executed by all of the components in a search handler chain before the request is processed. By specifying that a non-standard component is in the “first-components” list – it will be executed before the query is sent to the index by the downstream QueryComponent. This gives these early components a chance to modify the query before it is executed by the Lucene engine (or distributed to other shards in SolrCloud). The QueryAutoFilteringComponent works by creating a mapping of term values to the index fields that contain them. It uses the Lucene UnivertedIndex and the Solr TermsComponent (in SolrCloud mode) to build this map. This “inverse” map of term value -> index field is then used to discover if any sub-phrase within a query maps to a particular index field. If so, a filter query (fq) or boost query (bq) – depending on the configuration – is created from that field:value pair and if the result is to be a filter query, the value is removed from the original query. The result is a series of query expressions for the phrases that were identified in the original query. An example will help to make this clearer. Assuming that we have indexed the following records: This example is admittedly a bit contrived in that the term “red” is deliberately ambiguous – it can occur as a color value or as part of a brand or product_type phrase. So, with the OOTB Solr /select handler, a search for “red lion socks” brings back all 16 records. However, with the QueryAutoFilterComponent, only 2 results are returned (4 and 5) for this query. Furthermore, searching for “red wine” will only bring back one record (11) whereas searching for “red wine vinegar” brings back just record 12. What the filter does is to match terms with fields, trying to find the longest contiguous phrases that match mapped field values. So for the query “red lion socks” – it will first discover that “red” is a color, but then it will discover that “red lion” is a brand and this will supercede the shorter match that starts with “red”. Likewise, with “red wine vinegar”, it will first find “red” == color, then “red wine” == product_type then “red wine vinegar” == product_type and the final match will win because it is the longest contiguous match. It will work across fields too. If the query is “blue red lion socks” – it will discover that “blue” is a color, then that “blue red” is nothing so it will move on to the next unmatched token – “red”. It will then, as before, discover that “red lion” is a brand, reject “red lion socks” which doesn’t map to anything and finally find that “socks” is a product_type. From these three field matches it will construct a filter (or boost) query with the appropriate mapping of field name to field value. The result of all of this is a translation of the Solr query: q=blue red lion socks to a filter query: fq=color:blue&fq=brand:”red lion”&fq=product_type:socks This final query brings back just 1 result as opposed to 16 for the unfiltered case. In other words, we have increased precision from 6.25% to 100%! Adding case sensitivity and synonym support: One of the problems with using string fields as the source of metadata for noun phrases is that they are not analyzed (as discussed above). This limits the set of user inputs that can match – without any changes, the user must type in exactly what is indexed, including case and plurality. To address this problem, support for basic text analysis such as case insensitivity and stemming (singular/plural) as well as support for synonyms was added to the QueryAutoFilteringComponent. This adds to the code complexity somewhat but it makes it possible for the filter to detect synonymous phrases in the query like “couch” or “lounge chair” when “Sofa” or “Chaise Lounge” were indexed. Another thing that can help at an application level is to develop a suggester for typeahead or autocomplete interfaces that uses the Solr terms component and facet maps to build a multi-field suggester that will guide users towards precise and actionable queries. I hope to have a post on this in the near future. Source Code For those that are interested in how the autofiltering component works or would like to use it in your search application, source code and design documentation are available on github. The component has also been submitted to Solr (SOLR-7539 if you want to track it). The source code on github is in two versions, one that compiles and runs with Solr 4.x and the other that uses the new UninvertingReader API that must be used in Solr 5.0 and above. Conclusions The QueryAutoFilteringComponent does a lot more than the simple implementation introduced in the previous post. Like the previous example, it turns a free form queries into a set of Solr filter queries (fq) – if it can. This will eliminate results that do not match the metadata field values (or their synonyms) and is a way to achieve high precision. Another way to go is to use the “boost query” or bq rather than fq to push the precise hits to the top but allow other hits to persist in the result set. Once contextual phrases are identified, we can boost documents that contain these phrases in the identified fields (one of the chicken-and-egg problems with query-time boosting is knowing what field/value pairs to boost). The boosting approach may make more sense for traditional search applications viewed on laptop or workstation computers whereas the filter query approach probably makes more sense for mobile applications. The component contains a configurable parameter “boostFactor” which when set, will cause it to operate in boost mode so that records with exact matches in identified fields will be boosted over records with random or partial token hits.

From the DevNation Agenda site: You've probably heard a lot about Linux containers and the exciting potential they hold. In this presentation, Matt Hicks will cover how Docker and Kubernetes have evolved to fundamentally change how you will approach development and operations. If you are looking for an understanding of the technology and how it relates to the common roles in IT today, this is the talk to watch. Speaker: Matt Hicks -- Vice President of engineering, Red Hat Matt Hicks is a founding member of the OpenShift by Red Hat team. He has spent more than a decade in software engineering, with a variety of roles in development, operations, architecture, and management. His real expertise is in bridging the gap between developing code and actually running it in production. An expert in IT and cloud-based architectures, he spends his time these days evolving OpenShift to use the power of cloud and make developers more productive.

Welcome to Useful PC Guide, we are covering latest technology news with many topics on computing, mobile, programming, technology, computer games, games, mobile games, Apple iOS, and Android apps as well as online tutorials, guides and how-to articles. UsefulPCGuide.com website also regularly updates new Windows OS tips and tricks to resolve your problems, as well as iOS and Android issues. You can read an example tutorial from us about how to fix your connection is not private error in Google Chrome in Windows OS. This guide will help you to learn more about causes of this error, and appropriate ways to troubleshoot the issues on your Chrome browser. Most of our tips and tricks are include images and very easy to read and follow up the instructions. Visit usefulguide.com for more good news and tutorials.

I don't often write about tools I use when for my daily software development tasks. I recently realized that I really should start to share more often my workflows and weapons of choice. One thing that I have a hard time enduring while doing Python code reviews, is people writing utility code that is not directly tied to the core of their business. This looks to me as wasted time maintaining code that should be reused from elsewhere. So today I'd like to start with retrying, a Python package that you can use to… retry anything. It's OK to fail Often in computing, you have to deal with external resources. That means accessing resources you don't control. Resources that can fail, become flapping, unreachable or unavailable. Most applications don't deal with that at all, and explode in flight, leaving a skeptical user in front of the computer. A lot of software engineers refuse to deal with failure, and don't bother handling this kind of scenario in their code. In the best case, applications usually handle simply the case where the external reached system is out of order. They log something, and inform the user that it should try again later. In this cloud computing area, we tend to design software components with service-oriented architecture in mind. That means having a lot of different services talking to each others over the network. And we all know that networks tend to fail, and distributed systems too. Writing software with failing being part of normal operation is a terrific idea. Retrying In order to help applications with the handling of these potential failures, you need a plan. Leaving to the user the burden to "try again later" is rarely a good choice. Therefore, most of the time you want your application to retry. Retrying an action is a full strategy on its own, with a lot of options. You can retry only on certain condition, and with the number of tries based on time (e.g. every second), based on a number of tentative (e.g. retry 3 times and abort), based on the problem encountered, or even on all of those. For all of that, I use the retrying library that you can retrieve easily on PyPI. retrying provides a decorator called retry that you can use on top of any function or method in Python to make it retry in case of failure. By default, retry calls your function endlessly until it returns rather than raising an error. import randomfrom retrying import retry @retrydef pick_one(): if random.randint(0, 10) != 1: raise Exception("1 was not picked") This will execute the function pick_one until 1 is returned by random.randint. retry accepts a few arguments, such as the minimum and maximum delays to use, which also can be randomized. Randomizing delay is a good strategy to avoid detectable pattern or congestion. But more over, it supports exponential delay, which can be used to implement exponential backoff, a good solution for retrying tasks while really avoiding congestion. It's especially handy for background tasks. @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)def wait_exponential_1000(): print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards" raise Exception("Retry!") You can mix that with a maximum delay, which can give you a good strategy to retry for a while, and then fail anyway: # Stop retrying after 30 seconds anyway>>> @retry(wait_exponential_multiplier=1000, wait_exponential_max=10000, stop_max_delay=30000)... def wait_exponential_1000():... print "Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwards"... raise Exception("Retry!")...>>> wait_exponential_1000()Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsWait 2^x * 1000 milliseconds between each retry, up to 10 seconds, then 10 seconds afterwardsTraceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 212, in call raise attempt.get() File "/usr/local/lib/python2.7/site-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/local/lib/python2.7/site-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "", line 4, in wait_exponential_1000 Exception: Retry! A pattern I use very often, is the ability to retry only based on some exception type. You can specify a function to filter out exception you want to ignore or the one you want to use to retry. def retry_on_ioerror(exc): return isinstance(exc, IOError) @retry(retry_on_exception=retry_on_ioerror)def read_file(): with open("myfile", "r") as f: return f.read() retry will call the function passed as retry_on_exception with the exception raised as first argument. It's up to the function to then return a boolean indicating if a retry should be performed or not. In the example above, this will only retry to read the file if an IOError occurs; if any other exception type is raised, no retry will be performed. The same pattern can be implemented using the keyword argument retry_on_result, where you can provide a function that analyses the result and retry based on it. def retry_if_file_empty(result): return len(result) <= 0 @retry(retry_on_result=retry_if_file_empty)def read_file(): with open("myfile", "r") as f: return f.read() This example will read the file until it stops being empty. If the file does not exist, an IOError is raised, and the default behavior which triggers retry on all exceptions kicks-in – the retry is therefore performed. That's it! retry is really a good and small library that you should leverage rather than implementing your own half-baked solution!

Written by Joel Bancroft-Connors for LeadingAgile. Okay, so your enterprise wants to start an agile transformation. Good for you! We’ll assume you know why you are doing it, what the values are and that it’s not an overnight process. That still leaves a question of where in the organization do you start? Do you start with a small scale team level approach? Do you get executive sponsorship for a top down push? Do you work through the PMO? And what about middle management? The answer is, yes. Let’s look at the various entrance vectors for an agile transformation, and why they can fail. From the Team Up When I first gained a formal understanding of agile (like many I’d been doing it for years without realizing it), my basic Shu understanding of agile was very team and individual focused. I think my background in customer service made this a very natural place to go to. As a natural extension of this I believed that “agile must grow from the teams”. If you believe you are agile, you will be. It was at this time I first came up with my “Better people lead to better teams, better teams to better projects, better projects to better products, better products leads to better companies and better companies will make a better world.” philosophy. Unfortunately, this is not unlike the kid with a blanket tied around his neck that jumps off his parent’s roof, in the belief he can fly. Belief will only carry you so far in the face of the law of gravity. A team level agile transformation can only go so far in the enterprise before it runs into the impediments of large organizations. From the Top Down At the other end of the spectrum you have agilists that firmly believe an agile transformation must come from the executive level. Without their support, you can never conquer the agile anti-bodies and organizational impediments. The most common problem with this method, is a failure to commit. The executive says “we’re going agile” and may even hire some consultants to come in and help. Only like the product manager who doesn’t get the shift to being a product owner, the executive does not take part in the transformation. Mandates and visions from the C-Suite rarely succeed unless the executives are willing to invest their time directly into the effort. Even if they do, they can run into strong resistance from the middle without constant support from the top. Meet in the Middle For a time I believed that this was the secret to success. Find a team that wanted to do an agile pilot and get the executive to support this from the top down. This too is fraught with risk. I learned this was not unlike burning the candle at both ends. Pretty soon the middle is melting. Even if the agile pilot was successful, two things would rise up to crush it. The first being most agile pilots are small scale, high performing projects that won’t scale across the organizational impediments. The other problem was that the managers in the middle had a tendency to become detractors out of sheer fear of how this would change their role. Which led me to to the realization that without middle management bought in and supporting, you could not be successful. This launched me on a quest to help educate managers on what it meant to be a manager in an agile organization. While teaching managers to move from managing tasks, to enabling their teams was certainly valuable, it was not the magic entry point to start a transformation. It did build on my “better people” belief in that I was helping managers to support their directs better, even if they were not doing agile development. That didn’t help me with finding the vector to start an agile transformation. The PMO My focus on better managers, combined with my PMI background, led me to explore driving an agile transformation from the program management office. I really thought I was on to something here. The PMO typically owns process or has a lot of influence on it. And as peers to the middle management can exert some strong influence with them. The problems though came from all directions. Teams have a somewhat understanding wariness for the “process of the month” from project managers. “These non-engineers want to tell us how to write software?” Next, while the PMO might be able to get an executive sponsor, more often than not that sponsorship extends only as far as the kick-off meeting. And while the PMO does own process, because agile calls for a fundamental change in how people managers interact with their directs, those managers are usually highly resistant. So the bottom, the top and the middle all have their challenges for originating an agile transformation. So what do we do? A Total Approach While I was exploring coaching better managers, LeadingAgile ‘s founders, Mike and Dennis began to realize that only a systematic approach would work to successfully transform an enterprise scale organization to agile. By establishing an agile structure, governance and metrics, a company could bring clarity to their requirements, accountability (and ability) to the teams and be able to measurably track progress through working, tested software. This approach doesn’t focus on just one approach vector. Instead it sets up an agile transformation plan from portfolio, through the program level (product owner teams) to the delivery level. When the agile pilot is done, it’s not a cutting edge XP practice or Lean Startup. Instead the pilot is testing the very first step the rest of the organization will also take. The executive sponsor is directly involved, much like a product owner should be. The managers not only know what is happening, they are directly a part of it and get the support they need to be able to support their teams, not drive them to a death march release. And of course the teams get the hands-on help to make a transition to a Shu level agile framework, the first step in a multi-legged journey of an agile transformation. Not unlike Agile itself When we talk about creating a stable agile team, we often use the slice of cake analogy. The Scrum team (to pick an agile framework) should have all the skills needed to release an increment of potentially shippable product. An agile transformation needs to be a slice of cake through the organization, with everyone an equal player in the transformation. When we talk about enterprise agile release ceremonies we have release planning, sprint planning and the standup. With an agile transformation, the portfolio is the release planning, the program is the sprint planning and the teams the daily standup. Conclusion If you want a successful enterprise-scale agile transformation, you can’t start at the top, the bottom, or the middle. You have to start all along the continuum, at the same time. And for me, it’s been a realization that my “better people, better teams” philosophy isn’t a “one leads to the next” progression scale. Instead you have to work with the company as a whole, to make all levels better, together. I still believe better companies will save the world and that’s what I’m doing when I help a company do an enterprise-scale agile transformation.

I’m still working my way through the exercises in Think Bayes and in Chapter 6 needed to do some cleaning of the data in a CSV file containing information about the Price is Right. I downloaded the file using wget: wget http://www.greenteapress.com/thinkbayes/showcases.2011.csv And then loaded it into R and explored the first few rows using dplyr library(dplyr) df2011 = read.csv("~/projects/rLearning/showcases.2011.csv") > df2011 %>% head(10) X Sep..19 Sep..20 Sep..21 Sep..22 Sep..23 Sep..26 Sep..27 Sep..28 Sep..29 Sep..30 Oct..3 1 5631K 5632K 5633K 5634K 5635K 5641K 5642K 5643K 5644K 5645K 5681K 2 3 Showcase 1 50969 21901 32815 44432 24273 30554 20963 28941 25851 28800 37703 4 Showcase 2 45429 34061 53186 31428 22320 24337 41373 45437 41125 36319 38752 5 ... As you can see, we have some empty rows which we want to get rid of to ease future processing. I couldn’t find an easy way to filter those out but what we can do instead is have empty columns converted to ‘NA’ and then filter those. First we need to tell read.csv to treat empty columns as NA: df2011 = read.csv("~/projects/rLearning/showcases.2011.csv", na.strings = c("", "NA")) And now we can filter them out using na.omit: df2011 = df2011 %>% na.omit() > df2011 %>% head(5) X Sep..19 Sep..20 Sep..21 Sep..22 Sep..23 Sep..26 Sep..27 Sep..28 Sep..29 Sep..30 Oct..3 3 Showcase 1 50969 21901 32815 44432 24273 30554 20963 28941 25851 28800 37703 4 Showcase 2 45429 34061 53186 31428 22320 24337 41373 45437 41125 36319 38752 6 Bid 1 42000 14000 32000 27000 18750 27222 25000 35000 22500 21300 21567 7 Bid 2 34000 59900 45000 38000 23000 18525 32000 45000 32000 27500 23800 9 Difference 1 8969 7901 815 17432 5523 3332 -4037 -6059 3351 7500 16136 ... Much better!

Written by Mark Pollack. Today, we are pleased to announce the general availability of Spring XD 1.2, Spring XD 1.1.3 and the release of Flo for Spring XD Beta. 1.2.0.GA: zip 1.1.3.RELEASE: zip Flo for Spring XD Beta You can also install XD 1.2 using brew and rpm The 1.2 release includes a wide range of new features and improvements. The release journey was an eventful one, mainly due to Spring XD’s popularity with so many different groups, each with their respective request priorities. However the Spring XD team rose to the challenge and it is rewarding to look back and review the amount of innovation delivered to meet our commitments toward simplifying big data complexity. Here is a summary of what we have been busy with for the last 3 months and the value created for the community and our customers. Flo for Spring XD and UI improvements Flo for Spring XD is an HTML5 canvas application that runs on top of the Spring XD runtime, offering a graphical interface for creation, management and monitoring streaming data pipelines. Here is a short screencast showing you how to build an advanced stream definition. You can browse the documentation for additional information and links to additional screen casts of Flo in action. The XD admin screen also includes a new Analytics section that allows you to easily view gauges, counters, field-value counters and aggregate counters. Performance Improvements Anticipating increased high-throughput and low-latency IoT requirements, we’ve made several performance optimizations within the underlying message-bus implementation to deliver several million messages per second transported between Spring XD containers using Kafka as a transport. With these optimizations, we are now on par with the performance from Kafka’s own testing tools. However, we are using the more feature rich Spring Integration Kafka client instead of Kafka’s high level consumer library. For anyone who is interested in reproducing these numbers, please refer to the XD benchmarking blog, which describes the tests performed and infrastructure used in detail. Apache Ambari and Pivotal HD To help automate the deployment of Spring XD on an Apache HadoopⓇ cluster, we added an Apache AmbariⓇ plugin for Spring XD. The plugin is supported on both Pivotal HD 3.0 and Hortonworks HDP 2.2 distributions. We also added support in Spring XD for Pivotal HD 3.0, bringing the total number of Hadoop versions supported to five. New Sources, Processors, Sinks, and Batch Jobs One of Spring XD’s biggest value propositions is its complete set of out-of-the-box data connectivity adapters that can be used to create real-time and batch-based data pipelines, and these require little to no user-code for common use-cases. With the help of community contributions, we now have MongoDB, VideCap, and FTP as source modules, an XSLT-transformer processor, and FTP sink module. The XD team also developed a Cassandra sink and a language-detection processor. Recognizing the important role in the Pivotal Big Data portfolio, we have also added native integration with Pivotal Greenplum Database and Pivotal HAWQ through gpfdist sink for real-time streaming and also support for gpload based batch jobs. Adding to our developer productivity theme and the use of Spring XD in production for high-volume data ingest use-cases, we are delighted to recognize Simon Tao and Yu Cao (EMC² Office of The CTO & Labs China), who have been operationalizing Spring XD data pipelines in production since 2014 and also for the VideCap source module contribution. Their use-case and implementation specifics (in their own words) are below. “There are significant demands to extract insights from large magnitude of unstructured video streams for the video surveillance industry. Prior to being analyzed by data scientists, the video surveillance data needs to be ingested in the first place. To tackle this challenge, we built a highly scalable and extensible video-data ingestion platform using Spring XD. This platform is operationally ready to ingest different kinds of video sources into a centralized Big Data Lake. Given the out-of-the-box features within Spring XD, the platform is designed to allow rich video content processing capabilities such as video transcoding and object detection, etc. The platform also supports various types of video sources—data processors and data exporting destinations (e.g. HDFS, Gemfire XD and Spark)—which are built as custom modules in Spring XD and are highly reusable and composable. With a declarative DSL, a video ingestion stream will be handled by a video ingestion pipeline defined as Directed Acyclic Graph of modules. The pipeline is designed to be deployed in a clustered environment with upstream modules transferring data to downstream ones efficiently via the message bus. The Spring-XD distributed runtime allows each module in the pipeline to have multiple instances that run in parallel on different nodes. By scaling out horizontally, our system is capable of supporting large scale video surveillance deployment with high volume of video data and complex data processing workloads.” Custom Module Registry and HA Support Though we have had the flexibility to configure shared network location for distributed availability of custom modules (via: xd.customModule.home), we also recognized the importance of having the module-registry resilient under failure scenarios—hence, we have an HDFS backed module registry. Having this setup for production deployment provides consistent availability of custom module bits and the flexibility of choices, as needed by the business requirements. Pivotal Cloud Foundry Integration Furthering the Pivotal Cloud Foundry integration efforts, we have made several foundation-level changes to the Spring XD runtime, so we are able to run Spring XD modules as cloud-native Apps in Lattice and Diego. We have aggressive roadmap plans to launch Spring XD on Diego proper. While studying Diego’s Receptor API (written in Go!), we created a Java Receptor API, which is now proposed to Cloud Foundry for incubation. Next Steps We have some very interesting developments on the horizon. Perhaps the most important, we will be launching new projects that focus on message-driven and batch-oriented “data microservices”. These will be built directly on Spring Boot as well as Spring Integration and Spring Batch, respectively. Our main goal is to provide the simplest possible developer experience for creating cloud-native, data-centric microservice apps. In turn, Spring XD 2.0 will be refactored as a layer above those projects, to support the composition of those data microservices into streams and jobs as well as all of the “as a service” aspects that it provides today, but it will have a major focus on deployment to Cloud Foundry and Lattice. We will be posting more on these new projects soon, so stay tuned! Feedback is very important, so please get in touch with questions and comments via * StackOverflowspring-xd tag * Spring JIRA or GitHub Issues Editor’s Note: ©2015 Pivotal Software, Inc. All rights reserved. Pivotal, Pivotal HD, Pivotal Greenplum Database, Pivotal Gemfire and Pivotal Cloud Foundry are trademarks and/or registered trademarks of Pivotal Software, Inc. in the United States and/or other countries. Apache, Apache Hadoop, Hadoop and Apache Ambari are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All Posts Engineering Releases News and Events

You will aggregate a lot of logs over the lifetime of your product and codebase, so it’s important to be able to search through them. In the rare case of a security issue, not having that capability is incredibly painful. You might be able to use services that allow you to search through the logs of the last two weeks quickly. But what if you want to search through the last six months, a year, or even further? That availability can be rather expensive or not even an option at all with existing services. Many hosted log services provide S3 archival support which we can use to build a long-term log analysis infrastructure with AWS Redshift. Recently I’ve set up scripts to be able to create that infrastructure whenever we need it at Codeship. AWS Redshift AWS Redshift is a data warehousing solution by AWS. It has an easy clustering and ingestion mechanism ideal for loading large log files and then searching through them with SQL. As it automatically balances your log files across several machines, you can easily scale up if you need more speed. As I said earlier, looking through large amounts of log files is a relatively rare occasion; you don’t need this infrastructure to be around all the time, which makes it a perfect use case for AWS. Setting Up Your Log Analysis Let’s walk through the scripts that drive our long-term log analysis infrastructure. You can check them out in the flomotlik/redshift-logging GitHub repository. I’ll take you step by step through configuring the whole setup of the environment variables needed, as well as starting the creation of the cluster and searching the logs. But first, let’s get a high-level overview of what the setup script is doing before going into all the different options that you can set: Creates an AWS Redshift cluster. You can configure the number of servers and which server type should be used. Waits for the cluster to become ready. Creates a SQL table inside the Redshift cluster to load the log files into. Ingests all log files into the Redshift cluster from AWS S3. Cleans up the database and prints the psql access command to connect into the cluster. Be sure to check out the script on GitHub before we go into all the different options that you can set through the .env file. Options to set The following is a list of all the options available to you. You can simply copy the .env.template file to .env and then fill in all the options to get picked up. AWS_ACCESS_KEY_ID AWS key of the account that should run the Redshift cluster. AWS_SECRET_ACCESS_KEY AWS secret key of the account that should run the Redshift cluster. AWS_REGION=us-east-1 AWS region the cluster should run in, default us-east-1. Make sure to use the same region that is used for archiving your logs to S3 to have them close. REDSHIFT_USERNAME Username to connect with psql into the cluster. REDSHIFT_PASSWORD Password to connect with psql into the cluster. S3_AWS_ACCESS_KEY_ID AWS key that has access to the S3 bucket you want to pull your logs from. We run the log analysis cluster in our AWS Sandbox account but pull the logs from our production AWS account so the Redshift cluster doesn’t impact production in any way. S3_AWS_SECRET_ACCESS_KEY AWS secret key that has access to the S3 bucket you want to pull your logs from. PORT=5439 Port to connect to with psql. CLUSTER_TYPE=single-node The cluster type can be single-node or multi-node. Multi-node clusters get auto-balanced which gives you more speed at a higher cost. NODE_TYPE Instance type that’s used for the nodes of the cluster. Check out the Redshift Documentation for details on the instance types and their differences. NUMBER_OF_NODES=10 Number of nodes when running in multi-mode. CLUSTER_IDENTIFIER=log-analysis DB_NAME=log-analysis S3_PATH=s3://your_s3_bucket/papertrail/logs/862693/dt=2015 Database format and failed loads When ingesting log statements into the cluster, make sure to check the amount of failed loads that are happening. You might have to edit the database format to fit to your specific log output style. You can debug this easily by creating a single-node cluster first that only loads a small subset of your logs and is very fast as a result. Make sure to have none or nearly no failed loads before you extend to the whole cluster. In case there are issues, check out the documentation of the copy command which loads your logs into the database and the parameters in the setup script for that. Example and benchmarks It’s a quick thing to set up the whole cluster and run example queries against it. For example, I’ll load all of our logs of the last nine months into a Redshift cluster and run several queries against it. I haven’t spent any time on optimizing the table, but you could definitely gain some more speed out of the whole system if necessary. It’s just fast enough already for us out of the box. As you can see here, loading all logs of May — more than 600 million log lines — took only 12 minutes on a cluster of 10 machines. We could easily load more than one month into that 10-machine cluster since there’s more than enough storage available, but for this post, one month is enough. After that, we’re able to search through the history of all of our applications and past servers through SQL. We connect with our psql client and send of SQL queries against the “events’ database. For example, what if we want to know how many build servers reported logs in May: loganalysis=# select count(distinct(source_name)) from events where source_name LIKE 'i-%'; count ------- 801 (1 row) So in May, we had 801 EC2 build servers running for our customers. That query took ~3 seconds to finish. Or let’s say we want to know how many people accessed the configuration page of our main repository (the project ID is hidden with XXXX): loganalysis=# select count(*) from events where source_name = 'mothership' and program LIKE 'app/web%' and message LIKE 'method=GET path=/projects/XXXX/configure_tests%'; count ------- 15 (1 row) So now we know that there were 15 accesses on that configuration page throughout May. We can also get all the details, including who accessed it when through our logs. This could help in case of any security issues we’d need to look into. The query took about 40 seconds to go though all of our logs, but it could be optimized on Redshift even more. Those are just some of the queries you could use to look through your logs, gaining more insight into your customers’ use of your system. And you et all of that with a setup that costs $2.50 an hour, can be shut down immediately, and recreated any time you need access to that data again. Conclusions Being able to search through and learn from your history is incredibly important for building a large infrastructure. You need to be able to look into your history easily, especially when it comes to security issues. With AWS Redshift, you have a great tool in hand that allows you to start an ad hoc analytics infrastructure that’s fast and cheap for short-term reviews. Of course, Redshift can do a lot more as well. Let us know what your processes and tools around logging, storage, and search are in the comments.

Written by Andrew Graves for LeadingAgile. People don’t buy what you do, they buy why you do it. – Simon Sinek Over the last few years I have come across the notion of an Agile Ecosystem and several different thoughts around what constitutes an Agile Ecosystem. I have seen people in the industry refer to the Agile Ecosystem as a number of practices used together to achieve a delivery team’s agility goals. For example, the marriage of Scrum, XP, DevOps and Kanban. I think leveraging the best of breed in all of these areas makes perfect sense. However, the Agile Ecosystem is much bigger and needs to support the business as a whole. To drive sustainable agility and continuous improvement, an ecosystem must be built and maintained at an enterprise level. So… how do you go about building a high performing ecosystem that can provide value and continually delight customers? Start With Why It starts with Why… In other words what are the business drivers for making the organizational transformation. Why are you transforming? It may be to increase your market responsiveness, improve quality or greater predictability. Whatever it is, it needs to be clear. When the why is understood and transparent, the trek toward an agile ecosystem that supports the entire enterprise can begin. The trek starts with defining the end-state vision of your transformation and putting structure, governance and metrics in place to support the transformation. When you have clarified why you are going through the transformation, the next step should include forming teams at the delivery, program and portfolio levels. Then setting up a clear governance model and a way to demonstrate measurable progress. When all of these things are in place, the journey toward enterprise agility can begin. With the end-state in mind and a shared understanding of why, it’s time to get clarity on the team structure so governance and metrics can be applied. In my next post, I will touch on the teams in a scaled enterprise agile ecosystem and how they need to work together to deliver value to the customer.

Mobile app development has become vital for enterprises as they look to support new devices (phones, tablets, wearables, etc.) for internal use while also reaching out to their increasingly mobile customers. This approach makes sense: According to a comScore report, the number of mobile Internet users outnumbered desktop ones for the first time at some point in late 2013, and has since achieved significant separation. Many companies have responded to this change by implementing bring-your-own-device policies and building mobile apps that complement their full websites, mobile Web presence and/or desktop applications. Watch out for pitfalls in mobile apps: General risks and the recent Starbucks example However, both BYOD policies and mobile app development require due diligence around cybersecurity if they are to be worthwhile. Safety starts with well-designed applications that are strongly authenticated, do not leak sensitive data and are safe from popular attack vectors like brute-force password guessing. Unfortunately, many apps still have a long way to go on these fronts. An early 2014 study from MetaIntell discovered that 92 percent of the top 500 most popular Android apps at the time created privacy risks due to data leakage. Wary of leaky apps as well as what kinds of information users put into them, enterprises have understandably been concerned about the impact of mobile apps on their operations and BYOD initiatives. Security is often the biggest barrier to effective BYOD, and justifiably so considering that barely more than 40 percent of employees are required to have a security tool installed, according to Webroot. To get a sense of what could go wrong with today's mobile apps, consider what recently happened to Starbucks. The company's app is a mainstay on many phones, and at one time it accounted for the bulk of all mobile payments made in North America. The issue that arose over the last few months involved unauthorized card reloads and apparent account hijackings. The causes may have been mixed, with poor password management on the part of users possibly exacerbated by exploitation of the app's auto-reload feature and an April 2015 outage of the coffee chain's point-of-sale systems. At the end of the day, Starbucks implemented additional security questions and has been urged to add two-factor authentication into the app to prevent erroneous transactions. Catching mobile app security issues with a test management solution As we can see, mobile app security is multifactorial, requiring best efforts on the parts of end users, developers and infrastructure/network providers. For enterprises, the best approach to ensuring long-term security is to catch potential vulnerabilities early and often with a test management system. A test management solution supports both automated and manual testing, and receiving updates in real-time offers you the ability to make important decisions once issues arise. Regardless of how many tests, sprints and projects your company is running, all of them should be conveniently viewed from a lone interface, enabling a single source of truth that keeps your mobile app development initiatives on track.

DataOps is becoming an important consideration for organizations. Why? Well, DataOps is about making sure data is collected, analyzed, and available across the company – i.e. Ops insight for your decision-making systems like Hubspot, Tableau, Salesforce and more. Such systems are key to day-to-day operations and in many cases are as important as keeping your customer facing systems up and running. If you think about it, today every online business is a data driven business! Everyone is accountable to have up to the minute answers on what is happening across their systems. You can’t do this reliably without having DataOps in place. We have seen this trend across our own customer base at Logentries where more and more customers using log data to implement DataOps across their organization. Using log data for DataOps allows you to perform the following: Troubleshoot your systems managing your data by identifying errors and correlating data sources Get notified when one of these systems is experiencing issues via real time alerts or anomaly detection Analyze how these systems are used by the organization Logentries has always been great at 1 and 2 above, and this week we have enhanced Logentries to now allow you to perform easier and more powerful analytics with our new easy-to-use SQL like query language – Logentries QL (LEQL). LEQL is designed to make analyzing your log data dead simple. There are too many log management tools that are built around complex query languages and require data scientists to operate. Logentries is all about making log data accessible to anyone. With LEQL you are going to be able to use analytical functions like CountUnique, Min, Max, GroupBy, Sort…A number of our users have already been testing these out via our beta program. One great example is how Pluralsight has been using Logentries to manage and understand the usage of their Tableau environment. For example: Calculating the rate of errors over the the past 24 hours e.g. using LEQL Count function Understanding user usage patterns e.g. using GroupBy to understand queries performed grouped by different users Sorting the data to find the most popular queries and how long they are taking Being able to answer these types of questions enables DataOps teams to understand where they need to invest time going forward. For example, do I need to add capacity to improve query performance? Are internal teams having a good user experience or are they getting a lot of errors when they try to access data? At Logentries we are all about making the power of log data accessible to everyone and as we do this we are constantly seeing cool new use cases when using logs. If you have some cool use cases do let us know!

Every now and then I get work done in the home office. I’ve written previously about my setup, but after churning out some solution design today, I sat back and really took some time to appreciate the workspace. I’m really pleased with the configuration, it’s probably the best setup I’ve had in years. The desk is a former QLD police desk from the 1940s, so it wasn’t built for modern computers – not a problem, the cables run down the back which is just a minor annoyance. The keyboard and mouse are gaming varieties so that they perform well – the old Sennheiser (RF) wireless headset has been with me since 2006 and still works very well. The wooden clock (recently reviewed) acts as external speakers, a Bluetooth receiver and has a built in microphone so it can be used as a hands-free option for conference calls. It also features Qi wireless charging capability and also features a thermostat. Under the second monitor is a HDD caddy which supports USB3, and features 4 bays which can be used in parallel. I try to keep the desk reasonably neat, and there’s plenty of space so it doesn’t get too cluttered. I have a nice view out the window to a small courtyard which gets early morning sun.

After a bit of a gap I’m continuing the my series from DevOps Days Austin. After Damon Edwards kicked off the event, Michael Cote of Pivotal took the stage. Cote presented “The coming donkey apocalypse — what happens when Devops goes mainstream.” Take a listen (you can find his slides below): <br> Some of the ground that Cote covers: What DevOps as a community needs to focus on next to expand Unicorns (eg Uber and Netflix), Horses (eg top banks) and Donkeys (mainstream organizations) 3 key areas of DevOps to focus on today Culture and process Supporting legacy code Tools and technology Interviews on tap: Cameron Haight – Gartner John Willis – Docker Paul Read – Release Engineering Approaches Extra Credit reading Here’s how we can push DevOps mainstream – Cote Opening Keynote – Damon Edwards Pau for now…

This post originally published in the AppsFlyer blog. A couple of weeks ago Nir Rubinshtein and I presented AppsFlyer’s data architecture in a meetup ofBig Data & Data Science Israel. One of the concepts that I presented there, which is worth expanding upon is “Data’s Hierarchy of Needs:” Data should Exist Data should be Accessible Data should be Usable Data should be Distilled Data should be Presented How can we make data “achieve its pinnacle of existence” and be acted upon? In other words, what are the areas that should be addressed when designing a data architecture if you want it to be complete and enable creating insights and value from the data you generate and collect. If done properly, your users might just act upon the data you provide. This list might seem a little simplistic but it is not a prescription of what to do but rather a set of reminders of areas we need to cover and questions we need answered to properly create a data architecture. Data Should Exist Well, of course data should exist, and it probably does. You should ask yourself however, is if the data that exists is the right data? Does the retention policy you have service the business needs? Does the availability fit your needs? Do you have all the needed links (foreign keys) to other data so you’d be able to connect it later for analysis? To make this more concrete, consider the following example: AppsFlyer accepts several types of events (launches, in-app events, etc.) which are tied to apps. Apps are connected to accounts (an account would have one or more applications, usually at least, an iOS app and an Android one). If we would save the accounts as the latest snapshot and an app changes ownership, the historical data before that change would be skewed. If we treat the accounts as a slowly changing dimension of the events, then we’d be able to handle the transition correctly. Note that we may still choose to provide the new owner the historic data but now it not the only option the system support and the decision can be based on the business needs. Data Should Be Accessible If data is written to disk it is accessible programmatically at least, however, there can be many levels of accessibility and we need to think about our end users needs and the level of access they’d require. At AppsFlyer, the data existence (mentioned above) is handled by processing all the messages that go through our queues using Kafka but that data is saved in sequence files and stored by event time. Most of our usage scenarios do have a time component but they are primarily handled by the app or account. Any processing that needs a specific account and would access the raw events would have to sift through tons of records (3.7+ billion a day at the time of this post) to find the few relevant ones. Thus, one basic move toward accessibility of data is to sort by apps so that queries will only need to access a small subset of the data and thus run much faster. Then we need to consider the “hotness” of the data i.e. what response times we need and for which types of data. For instance, aggregations, such as retention reports need to be accessed online (so called “sub-second” response), latest counts need near real-time , explorations of data for new patterns can take hours etc. To enable support of these varied usage scenarios, we need to create multiple projections of our data, most likely using several different technologies. AppsFlyer stores raw data in sequence files, processed data in parquet files (accessible via Apache Spark), aggregations and recent data in columnar RDBMS and near real-time is stored in-memory. The three different storage mechanisms I mentioned above (Parquet, columnar RDBMS and In-Memory Data Grid) used in AppsFlyer all have SQL access; this is not by chance. While we (the industry) went through a short period of NoSQL, SQL or almost-SQL is getting back to be the norm, even for semi-structured and poly-structured data. Providing an SQL interface to your data is another important aspect of data accessibility as it allows expanding the user base for the data beyond R&D. Again, this is important not just for your relational data… Data Should Be Usable What’s the difference between accessible data and usable data? For one there’s data cleansing. This is a no-brainer if you pull data from disparate systems but it is also needed if your source is a single system. Data cleansing is what traditional ETL is all about and the techniques still apply. Another aspect of making data usable is enriching it or connecting it to additional data. Enriching can happen from internal sources like linking CRM data to the account info. This can also be facilitated by external sources as with getting the app category from the app store or getting device screen size from a device database. Last but not least, is to consider legal and privacy aspects of the data. Before allowing access to the data you may need to mask sensitive information or remove privacy-related data (sometimes you shouldn’t even save it in the first place). At AppsFlyer we take this issue very seriously and make major efforts to comply when working with partners and clients to make sure privacy-related data is handled correctly. In fact, we are also undergoing independent SOC auditing to make sure we are compliant with the highest standards. To summarize, to make the data usable you have to make sure it is correct, connect it to other data and you need to make sure that it is compliant with legal and privacy issues. Data Should Be Distilled Distilling insights is the reason we perform all the previous steps. Data in itself is of little use if it doesn’t help us make better decisions. There are multiple types of insights you can generate here beginning from the more traditional BI scenarios of slice and dice analytics going through real-time aggregations and trend analysis, ending in applying machine learning or “advanced analytics”. You can see one example of the type of insights that can be gleaned from our data by looking at theGaming Advertising Performance Index we recently published. Data Should Be Presented This point ties in nicely with the Gaming Advertising Performance Index example provided above. Getting insights is an important step, but if you fail to present them in a coherent and cohesive manner then the actual value users would be able to make of it is limited at best. Note that even if you use insights for making decisions (e.g. recommending a product to a user) you’d still need to present how well this decision is doing. There are many issues that need to be dealt with from UX perspective both in how users interact with the data and how the data is presented. An example of the former is deciding on chart types for the data. A simple example for the latter is when presenting projected or inaccurate data it should be clear to the users that they are looking at approximations to prevent support calls on numbers not adding up. Making sure all the areas discussed above are covered and handled properly is a lot of work but providing a solution that actually helps your users make better decisions is well worth it. The data’s hierarchy of needs is not a prescription of how to get there, it is merely a set of waypoints to help navigate toward this end goal. It helps me think holistically about AppsFlyer data needs and I hope following this post it would also help you. For more information about our architecture, check out the presentation from the meetup: Architecture for Real-Time and Batch Big Data Analytics Distilling insights @ AppsFlyer

lean startup is so overrated and trendy. all these acronyms like mvp, arrrr, p&l just get your head in the wrong place. who wants to be pirate anyway? they all had scurvy. i think lean startup causes more problems than it solves. i’d rather run a fat ( phat ) startup. i’ve done it many times before, and i liked it while it lasted. so, i’m going back to doing that. my favorite reason for running a phat startup was that i could delude myself. as much as i wanted. i could paint visions in my head, without actually needing to think. i think self-delusion is a great experience. somebody should productize that. unparalleled.why hire a pr company? you can spin your own yarns, since you believe so much more in your own vision than some–hired guns. yes, delicious delusions are the best, with lots of delicious money. you don’t need to confront day-to-day reality then. or learn anything. just execute. speaking of delicious, the next reason i like running a phat startup is the hors d’oeuvres at cocktail parties and events. if you’re running a phat startup, you can go to these events and hand out designer’y business cards. those cards have your name, phone number, and the title “founder” on them, or even better, “co-founder”. that means you’re part of a hip team.it’s lots of fun to show off and make friends, even though they aren’t your customers, in your industry, or potential funding providers. i still hang out with some of those friends from my phat days…even became a godfather for one of those friend’s children. another reason to create a phat startup: you can build anything you want. ignore those pesky customers and what they want. after all, who are they to tell you what you should build? what do they know?customers or prospects have a well-meaning way of deflating your vision. they all say they want something different than what you do. i mean, they’re good people and all, it’s just better to avoid them when you’re working on your idea. you wouldn’t want them to screw it up. on to the next part. since you’re out building your idea, you get to play with lots of techie toys. that’s my favorite part of being a phat founder. nobody tells me what technology i can’t play with. i can sign up for lots of websites and software. i can tweak things so that i really get my product to be polished and perfect before showing it to anyone.even though i probably spent a lot of time playing xbox as a phat founder, nobody needs to know that. my cocktail friends thought i was trying out the latest javascript framework, the one that makes coffee and writes a book for you at the same time. if you’re running a phat startup, you can just focus on raising investor and shareholder money. fund a lavish startup lifestyle. if you run out, you can always raise more. diluting your stake doesn’t really matter, as long as you can find someone else to give you money.some founders just don’t get it. why try to keep as much equity as you can? i mean, you wouldn’t want to be risking your savings or anything. that’s not the point here. you want to be living the good life now, not after your exit. so that’s my case for a phat startup. i hope you see that i’m onto something, and that these lean startup guys have it all wrong. stop getting canvassed. go phat. [image: kevin harber]

yesterday i was speaking with a gaggle of early stage tech entrepreneurs about lean startup. they were eager to learn more about about hypothesis testing. newbies wanting to learn about the lean startup approach. yet they were falling prey to what i call the “solutionizing bias”. they only wanted to talk and think about solutions. and making sales. it’s so tempting, easy and natural for founders to fall for that trap. as entrepreneurs, we’re naturally optimistic go-getters. we have a solution for every problem. we want to help out. yet when you’re building a new product, you don’t know whether your solution is important for your customer. if you don’t prove that first, then brace yourself for a long uphill battle. your solution solves a problem your customer has. the customer cares about their problem, not your solution. before you think about solutions, you need to know whether the problem your solution solves is important. in your customer’s eyes. that’s why validating your product idea is so important. first ensure that the problem you’re addressing affects a large percentage of your target market. if you focus on your solution only, you won’t know which problem is worth addressing. it’s blinding. imagine you’re reading a market research report about the biggest problems of your target market. in that report, there is a pie chart. let’s say your prospects have 4 different problems: a, b, c, d. 40% consider a their main concern, 30% b, 20% c, and 10% d. yet, you’ve already built a solution. you realize it addresses problem d. for the same amount of marketing effort, you’ll get 4 times less results than another founder who addresses problem a. in my experience with building products, i’ve managed to build products that addressed problem e. in other words, it was a problem i thought the market should have, but actually didn’t. i was dead in the water. i write about an example like that, and how to prevent it from happening to you, in launch tomorrow . while it’s critical to think in terms of sales you’re going to make when you start a business, first check if you are addressing a meaningful problem first. one that a big chunk of your market thinks they have. and that they’re willing to pay for a solution. then you set yourself up for hockey-stick growth. only then do you build a money-making machine. otherwise, you might as well be a missionary. if you follow the one-day launch sequence in launch tomorrow , you’ll get a decent blueprint to do exactly that. build what your customers want.

AMD and Intel have competed for years in the performance arena, and a recently announced CPU-GPU combo could make AMD the frontrunner. The high-performance CPU-GPU megachip will reportedly be able to deliver teraflops of performance. While AMD and Intel have sparred for supreme champion on the performance circuit, Intel’s flagship i7 5960x is supposedly the most powerful CPU available. Forbes contributor Jason Evangelho posted benchmarks of AMD’s R9 390 GPU compared to Nvidia’s GTX 980 Ti, and according to his research, the 390 beat the 980. So AMD appears to be dominating graphics performance. Lately, AMD has concentrated on delivering energy efficient components, and surprisingly hefty performance. I’m still amazed at the recent games my laptop, equipped with an AMD A10-4600k and Radeon 7660g integrated graphics, can run on high detail on a 1080p display. However, the forthcoming and yet unnamed CPU-GPU megachip won’t be aimed at gamers. Rather, it’s more tailored to workstations requiring beefy computing power, or servers. This doesn’t mean that a gaming variation won’t be available eventually, or that AMD’s megachip can’t be tweaked to fit gaming needs, but it’s made with computation in mind. This is massive news for high-performance PCs, and a migration toward CPU and GPU integration might be the next big advancement in computer hardware.

We’re excited to share a new infographic on the importance of social media for customer support. As more consumers are looking online to resolve customer service issues, companies are increasing their investment in social media. Adopting a social customer care policy isn’t optional anymore – it’s critical to staying competitive and relevant. Click on the image below to open a larger version of the infographic for easy viewing. You can also download the eBook: “Social Customer Care: How to Use Social Media to Improve Customer Support.” Like this post? Click here to subscribe to our blog and receive the latest content on social learning, customer support, sales enablement, or all three.

At last week’s New Relic User Group meetup in Chicago, David M. Kent, Senior Director, Technical Architecture, of giant food distributor US Foods told the “story of how New Relic introduced us to the concept of DevOps.” It’s a fascinating tale of transforming technology at a 162-year-old company, but let’s take a moment to put his presentation in context. Sean Carpenter, Product Mgr., New Relic David’s talk was the highlight of a rewarding evening for some 50 New Relic users that also featured a spirited round of “Stump the Relics” Q&A as well as our own Sean Carpenter sharing tips on how to manage complexity with New Relic’s new Service Maps. New Relic has a definite outlook on how to monitor, Sean said: The key is to start by focusing on providing a good experience, not tossing in a zillion features into version 1.0. Then you can work on continuous improvement as you learn more about what makes the most difference to actual users. Step one in US Foods’ quest for DevOps Formed in 1853 to sell provisions during the California Gold Rush, US Foods launched its first ecommerce site in 1999, and now books more than 50% of its annual e-commerce revenue from more than 107,000 active customers. But after deploying Release 2 of its e-commerce platforms in 2012, the company made the decision to rethink its e-commerce strategy, adopting an agile methodology as it moved toward Release 3 on a completely new architecture stack. (The company is now 40% done moving customers to the new platform, David said, after two years of development and planning.) David Kent, US Foods For Release 3, the company decided all new servers would be virtual machines, and brought in “agile coaches to teach us how to be agile,” David said. The team worked on automating their build, deployment, and testing processes. The long-term goal? To create a DevOps model comprised of building a culture of collaboration where dev, infrastructure, and QA teams work together harmoniously using advanced tools to automate traditionally manual processes. Overcoming challenges The biggest challenge the company faced, David noted, was the mindset of longtime managers on the operations side. As one of his slides noted: Some business apps written 30 years ago are still in production Some managers hired 30 years ago are still in production “We wanted to refocus on culture and help people understand why DevOps is a good idea [but] they still don’t understand why we needed to go faster… ‘We’ve been doing fine, meeting the business needs for 30 years, why do I need to deploy daily or weekly?’ they’d ask.” “We’re still trying to figure out how to fix that,” he said. “We are also trying to improve collaboration across teams with more video conferencing and collaboration tools.” But in a big, geographically distributed company (US Foods has its data center in Phoenix while its business development team is located in the Chicago suburbs), he believes travel is still important to “get face time with key people.” The role of New Relic David Kent presents to the Chicago User Group audience. US Foods brought in New Relic late in 2012: “We were having problems,” David said, and “we didn’t know how to troubleshoot them efficiently.” Although originally targeted to provide end-to-end transaction monitoring for Release 3, David said, US Foods has found surprising ways to get value out of New Relic in the meantime. “Any time an app was having performance problem, we said, ‘Hey, let’s put New Relic on it,’ and we can find out what the problem is.” Broader visibility is another nice thing about New Relic, David said. US Foods now has 121 people with visibility into all of these monitors. “They get the same performance information that the engineers have!” Going forward, the company is running a proof-of-concept trial with New Relic Mobile on the latest version of its mobile app, and is very interested in using New Relic Insights to incorporate nontechnical data. They want to track things like how many cases they’re shipping, David said. There’s a role for New Relic Synthetics, too. “Most of our traffic comes during the day,” David said. “Our customers are primarily ordering around noon… there’s no activity at night. But we also want to know if there’s a problem at night before our customers find it, and Synthetics can help with that.” It was a great presentation and a great night. Big thanks to Sprout Social for hosting everyone in their awesome space! DevOps Days discount Finally, for more on DevOps in Chicago, Jerry Cattell told user group attendees thatDevOpsDays is returning to Chicago on August 25 and 26. The conference brings development and operations together to solve problems, explore tools, and think big about DevOps concepts and philosophies—and New Relic users can get a 10% discount with the code NEWRELIC10. Also note that the call for proposals is open until June 30 if you want to share something about your company culture, development practices, deployment pipelines, metrics, monitoring, or interesting ways you are using New Relic. Join our Meetup groups and stay connected New Relic User Groups are a great opportunity to meet and learn from other New Relic users in your area, hang with the New Relic team, and get answers to lingering questions. If you’re in Chicago, be sure to join our Chicago New Relic User Group Meetup.com page and get notifications for future events. To see all the New Relic User Group events coming up, visit our Events page (just choose “NRUG” from the pull-down menu to see only the User Group events). Interested in speaking at our next event or setting up a New Relic User Group in your city? Drop us a note at [email protected]. Note: Event dates, speakers, and schedules are subject to change without notice.