Don’t Try This at Home: The Dangers of DIY Toolchain Integration
Like the plumber, it’s crucial to make sure you find the right way to go about integrating your toolchain.
Join the DZone community and get the full member experience.Join For Free
Remember that time when your partner flagged a leak under the sink and suggested calling a plumber? "How hard can it be," you asked. A few grunts and speculative twists and bangs (and a few choice words) later, not to mention a generous application of duct tape, the leak had stopped. Disaster averted (at least until the problem resurfaced a few weeks later). And it was even worse that time. The kitchen flooded, causing more damage than Vesuvius.
DIY, unless you genuinely know what you’re doing, always comes at a cost. Sure, you can learn to become a better plumber. But plumbing is a specialty — and an expensive service — for a reason. On average, it takes nearly a decade to learn the ins and out of the trade.
After all, plumbing isn’t just about fixing emergency issues; it’s about assessing a customer’s plumbing infrastructure to minimize future risk (and costs) that could devalue a home and hammer the wallet. The same mentality applies when integrating the tools that plan, build, and deliver software at scale. Except in that scenario, you risk upsetting your customers, your business, your software delivery teams, your CFO, and your CEO. You even risk your livelihood.
So, like the plumber, it’s crucial to make sure you find the right way to go about integrating your toolchain.
What’s All the Fuss About Integration?
The benefits of connecting the tools, teams, and processes in your software delivery value stream are well documented. Many of the Fortune 100 are accelerating business value delivery by automating, visualizing and measuring their software delivery value streams. They’re doing this by connecting the people and the technology that carry out the work across key stages by investing in enterprise-grade integration that is built for a purpose.
By connecting these stages in the process from ideation to operation and back through the customer feedback loop, organizations can automate the flow of product-critical information that teams need to collaborate in real time. This seamless flow enables them to continuously deliver and improve the software products that are giving their companies an edge. Benefits include:
Eliminating tedious and time-wasting manual overhead such as email, IMs, meetings, tool switching, duplicate entry, and spreadsheet management
Allowing specialists to access the information they need to do their job in their tool of choice
Accelerating the rate of delivery
Enhancing cross-team collaboration and productivity
Improving time to value (TtV) and mean time to resolution (MTTR)
Enabling end-to-end visibility to spot bottlenecks and opportunities for optimization
Creating end-to-end traceability for compliance and continuous improvement
The Challenges of DIY
Large-scale organizations often severely underestimate the complexity of DIY solutions, how long they take to build, and how difficult it is to keep them up-to-date. Key issues they face include:
Dealing with public APIs and each tool’s limitations
Developing bi-directional sync across two endpoints
Meeting sophisticated and nuanced uses cases
Account for project idiosyncrasies in tools, workflows, and processes
Staying on top of new tool versions and API changes
Updating integration as fields, values, and workflows change
Several FTEs to develop and maintain
Frequent outages due to new versions, API changes, and infrastructure changes
Gradual rollout, delayed benefits
Long lead times to implement necessary changes
It's easy to understand why you may conclude that building your own solution is the most cost-effective. Sounds simpler — harness the resources you already have, and remove the headache of outsourcing and the baggage that sometimes comes with it. But the costs involved in DIY integration are eye-watering. There are the opportunity, quantitative, and qualitative costs to bear in mind — costs that can amount to millions of dollars a year.
The Costs of DIY Integration
A reliable, scalable, and adaptable integration product is like any function-specific software product you invest in for your business. Would you build your own CRM, business intelligence, or accounting software? Better still, would you build your own Agile planning or test management tool?
“But we’ve invested heavily in our development teams, isn’t that what we’re paying them for,” you may ask. A fair question. Consider, however, the business reason why you invested in your development teams in the first place. Was it for them to be spending countless hours to understand the complex intricacies of integrating tools (and constant maintenance of the integrations once they’re deployed) rather than doing the work they were hired for?
The fact is your scarce and highly skilled development resources should be focusing on one thing and one thing only — delivering value to customers. They should be the beneficiaries of toolchain integration, not the creators.
To create robust integrations, developers must be able to understand the requirements of the integration by performing a rigorous technical analysis of the two endpoints, a business analysis of how they’re used, and by using their APIs to create and test the integrations on an on-going basis. These are typically not junior developers. If your top-level developers and testers are building and maintaining these integrations, it’s likely that they are taking a considerable amount of valuable time away from the work that differentiates your organization from its competitors.
These costs are related to the quality of the solution:
Will it support your use cases and customized endpoints?
Does it have a negligible impact on the performance of the endpoints?
Will the difficulty in adding new endpoints, or new versions of existing endpoints, limit the benefits of integration?
When change happens, will it require extensive re-coding?
Does it operate with high-reliability?
Can it handle large deployments?
It may be possible to create a minimum viable product (MVP) as the first version of integration, but building the kind of integration that actually supports the way your organization and its tools operate may be quite another thing, especially as your business evolves and scales.
Whether buying a contract phone or investing in a solution that will help you generate more business value from software faster, we must always consider the total cost of ownership (TCO).
Integration is about implementing the foundational hub through which product-critical information continuously flows between those who need it.
This rich information is what determines how well and how fast you’re delivering value across an increasingly complex network of specialist teams, tools, and processes. For that hub to be sufficient, it must be automated, reliable, scalable, and flexible.
If you’re working for an enterprise, it’s highly likely you’re looking to integrate more than just two tools (say development to test). That’s just one sliver of the software delivery value stream. An analysis of over 300 global organizations found the following:
Multiple integrations are required to automate the flow of work beyond Agile and DevOps and connect and automate the flow of value from end-to-end.
With that in mind, the TCO must incorporate costs that relate to designing, developing, testing, and maintaining the multiple integrations that support the software delivery value stream during the lifespan of the product, which could be decades.
For the sake of illustration, the following scenario includes:
Developers using an Agile tool (like IBM RTC, Microsoft TFS, JIRA, Rally/CA Agile Central, VersionOne)
Testers using a testing tool (like HPE QC) over a three-year timeframe, with one HPE QC update and two Agile tool updates applied (Agile tools typically have 2-4 updates per year)
As the previously mentioned analysis into the value streams of 300 organizations demonstrates, common scenarios involve integrating more than two tools and require more frequent upgrades, driving even higher TCO differentiation.
The calculations for the following cost comparisons are based on an enterprise’s typical integration scenario using the following formula:
Two Years Is Too Long
The estimated 12 to 24 months to set up and maintain an integration between "just" an Agile and test/QA tool is just too long. Such timeframes are not conducive to accelerating the time to value of an enterprise's products. Nor is the sky-high cost of nearly $900,000 just to flow information between just one stage of the value stream.
So before undertaking a DIY integration, consider other approaches, such as purpose-built integration infrastructures that are geared toward scaling software delivery and include “already built” solutions that connect your software delivery teams.
Not only are you likely to save a lot of time and money, but you will also implement a steadfast solution that will grow and adapt to the software needs of your business, maximize ROI, and liberate your highly skilled developers to focus on driving innovation and generate even more business value.
Opinions expressed by DZone contributors are their own.