In the beginning there was a simply acronym: MTTI (mean time to innocence). Weary after years of costly and time-consuming warroom battles, IT organisations turned to APM solutions to give an objective application-level view of production incidents. As a result, application issues are swiftly pinpointed and fixed, accelerating time to repair by up to 90%.
In fact, gravitation towards fact-based constructive issue management spawned a whole new movement – DevOps – with the goal of ingraining this maturity and cooperative spirit into IT organizations from the ground up. The movement was discussed by Jim in a previous blog post. Of course, easily accessible fact-based information about application behavior in production is a necessary prerequisite to this.
Looking back, before DevOps or even MTTI were topical buzzwords, this basic ability to foster communication between teams proved to be an invaluable benefit to the more drab and well-worn business realities of offshoring and outsourcing.
This blog reviews 3 real-life examples of this:
- Managing external offshore development organisations
- Facilitating near shore development teams
- Bringing external developments in-house
Managing external offshore development organisations
Some months ago, I did some work with a then prospect who had started a trial of an APM solution.
When I spoke to them, they were delighted with the visibility that the APM solution provided out of the box for their .NET application, a SaaS Learning Management System.
Digging into what had sparked their interest in the use of APMs, they told me they had commissioned an outside development firm to rewrite their flagship application, which was somewhat dated and not architecturally fit to support some newer services the business wanted to offer to customers.
The good news was the new version of the app was live, and supporting around 10% of their existing customer base. The bad news? This 10% used the same hardware footprint as the remaining 90% on the old system. Extrapolating this hardware requirement for the entire user base would not only require a new datacenter, but also entirely break the business model for the application from an operational cost perspective.
For months prior to trying the APM solution, the external developers had been under huge pressure to optimize the application footprint (and some pretty lackluster performance too). Armed with only windows performance counters and intuition, weeks had been spent optimizing slow database queries, which only turned out to be 5% of the errant response times at a transaction level.
Having put the APM solution in place, the prospect
- Easily found specific application bottlenecks, allowing them to focus developers on high-impact remediation
- Could verify the developers had made the required improvements with each new release
Clearly, huge benefits at a technical level.
At a higher level, this helped lead to a more constructive relationship between the development shop and their customer – moving things away from the edge of litigation, constant finger-pointing, and blame shifting.
Facilitating near shore development teams
Another group I have worked with recently are responsible for a settlement system within a large global investment bank based in London. The system is developed in-house, and typical with most financial services institutions, the actual development team itself is located ‘near-shore’ in Eastern Europe to cut costs. The development processes are Agile, with new releases every few weeks.
Inevitably, with new releases can come new production issues and – of course – the best people to deal with these during the “bedding in” period are the developers themselves.
Another thing that is very common in the financial services industry is regulation, and this poses a problem in this scenario. Nobody is permitted to directly access the production systems from outside the UK due to data privacy regulations.
This means hands-on troubleshooting must be left to the on-shore architect staff who are not only expensive, but are not as well-equipped as the developers themselves to dig in to the issues in new code.
Enter an APM solution. The APM agents deployed in production made all the performance data readily available to anyone with the appropriate credentials, but – critically – having access to this does not expose ANY business data from the production system. Now, the near-shore development team can look directly at the non-functional behavior of their code in production, eliminating the time spent gathering sufficient log data to enable reproduction of issues in test environments. Bingo, the business case for the APM purchase is made!
There is an interesting side note to this, which applies much more widely too. Many customers have observed an “organic” improvement in service levels once an APM solution is installed in production. For the first time, developers can see how their code is actually working in the wild. Developer pride kicks in and suddenly non-functional stories are added to development backlogs to fix latent issues that get observed, which would have previously have gone unnoticed.
Bringing external developments in-house
Of course, as we all know the only constant in life is change, so no outsource is a one-way journey. As a result, I have come across several organizations that are now working on projects which were previously outsourced. Of course, once these customers have completed the initial challenge of recruiting a new development team they then need to get their arms around the existing codebase. Usually handover workshops can help with this, but in many cases these systems have been out- and in- sourced several times, with many changes of personnel along the way. There is only so much you can distill onto a whiteboard in a brain dump session, however long and well-intentioned.
It is here where the high-level visibility that APMs provide can be invaluable. Out of the box, APM solutions instrument previously unseen systems, automatically detecting and following transactions and draws up flow-maps. The end-to-end visibility of the entire system greatly eases the process. In fact, this system overview (and the ability to view how it changes over time) has proved invaluable for many customers for a number of reasons beyond whole-scale in (or out) sourcing, such as onboarding new team members, verifying compliance with architectural governance of externally developed code changes and so forth.
In summary, APM does not have to be all about troubleshooting and MTTI. Nor even necessarily about DevOps and brave new worlds. The easily configured deep insight that we provide into the dynamic behavior of your applications has many uses – and business cases – beyond the traditional MTTI/MTTR domain. APM is, after all, just one use-case (albeit an important one) for our Application Intelligence Platform.