The Impact of Technical Ignorance

I knew a Chief Software Architect from a major financial organization who was an anomaly: he had never developed software professionally. His undergraduate degree is in Accounting and Finance, and his most hands-on technology role was as a DBA. [His LinkedIn profile lists an early Programmer role, though he insisted he didn’t.] Even so, he was well-respected within his organization for his thought leadership and solutions, but nevertheless, it seemed an unusual career path. Since I last worked with him, he has moved into C-level roles at other organizations, confirming his abilities as a technology leader.

Then I thought of others I have worked with who are non-technical but positioned to impact technical direction and realized their lack of understanding impacted (and continues to impact) the quality of the software solutions we, as engineers, are expected to deliver.

Chief Non-Technical Officer

This CTO has been with her/his company for many years in many roles: Director of Support, Chief Strategy Officer, Chief Cultural Officer, and Chief Technical Officer. S/he does not deny that s/he is not a strong technologist – and at times a badge of honor – yet confidently states decisions and direction that they become a fait accompli: alternatives that challenge her/his understanding are not often well received. At times, her/his inner circle helps to form a more nuanced understanding, but only to a point: overcoming her/his existing preconceived notions is difficult, and blatant opposition results in being sidelined from future discussions. By no means is s/he a total technical novice, but fundamental change requires extensive effort and time.

Her/his oft-repeated mantra went something like this:

Don’t tell me you’re refactoring; refactoring brings no value to our customers.

Harking back to her/his strategy days, where feature-feature-feature is the overwhelming driver, this mantra confirmed her/his denial or lack of understanding of the current state of the product. The growing and maturing customer base made clear that areas of the product needed love and attention, but proposed efforts to address them were not prioritized because – in her/his view of the world – there was no visible benefit to their customers, at least when focused on customers asking for new or extended features.

The real technologists of the company understood the potential benefits to both the customer and company: performance and scaling improvements, reduced cloud costs, faster deployments, fewer outages, faster feature delivery, reduced technology stack, and consistent and intuitive user experience. Regardless of potential benefits, nothing called out as refactoring would survive planning. The problems continued to grow, and the problems continued to be ignored. Sigh.

Product

To be clear, I have no interest in becoming a product owner: the wide-ranging responsibilities require a breadth of knowledge and experience not often found in a single person, while their many stakeholders – both internal and external – have contradictory goals and agendas that need to be balanced. I view it as a political role, finding compromises that please (appease) most, with no one getting everything s/he desired. This role is not for the weak and timid.

Once we accept that product owners are unlikely to have the background or experiences necessary to handle all responsibilities, we can then understand why the focus is on those responsibilities understood or deemed important by their leaders. Outside of organizations offering technical solutions, product owners often have a stronger business understanding than technology understanding based on their work experience. Perhaps not surprisingly, the product is defined by business expectations more so than technical requirements: future features and functionality are defined by understanding industry trends, reviewing customer feedback, interpreting — sales and usage analytics, defining the user experience, etc. In essence, the product owner is an overclocked business analyst.

Real-World Example

A particular product manager focused only on rapidly releasing new features regardless of technical stability. Over time, the issues rose to the point where outages – not processing failures, actual outages – occurred daily and could no longer be ignored. She continued to view the work as unnecessary and not beneficial to the product, resulting in this exchange during quarterly planning:

The result is product owners often eschew – whenever possible – technology and technical viability aspects of the product, reducing the impact of technology during product planning. Instead of top-down planning, individual engineers attempt to push technical issues bottom-up, which is very difficult and often unsuccessful. Organizations require a strong engineering discipline and culture to offset the business focus of product owners, but it remains a frustrating challenge.

[Of course, production technology issues do arise that demand immediate attention, but the resulting work is stressful, particularly for the engineers who are responsible for implementing the changes required; the result is often a one-off effort rather than fundamentally changing the overall culture.]

The Not-Ready-For-Prime-Time Implementation 

This is less about an individual or role but rather an organizational culture problem: proof-of-concepts assumed to be production-ready.

Software proofs-of-concept (POCs) are created to test new business concepts or determine the usefulness or applicability of new technology. POCs should be created with minimal engineering rigor that allows a quick and cheap implementation to be discarded without guilt once the results are evaluated. Most important, it is not intended to be a workable product.

Despite these clear expectations, too often, I’ve seen the business get excited at seeing the POC and want it available to customers immediately. The POC might be slightly enhanced or it might be unaltered, but it’s out there for the world (internal or external) to use. And when the problems start appearing – because, by definition, it was not intended for real-world usage – the finger-pointing begins.

Agile advocates snigger and say You needed an MVP, silly! but my experiences are much the same as POCs: poor. By definition, an MVP is a complete application without the bells and whistles, but corners are inevitably cut: crawling (of crawl/walk/run paradigm) when current volumes require walk, run, or even fly; minimal/non-existent observability; non-standard user experience; incomplete or incorrect API definitions; security through obscurity; incomplete error handling. When leaders decide to move forward after a successful MVP, the expectation is to expand and enhance the MVP implementation; in fact, it may be better to start over.

[I am not disavowing MVPs’ usefulness but rather am clarifying that organizations misuse/abuse the term and are, in fact, creating glorified POCs that are not complete, are not ready for users, and are not production ready. Just saying…]

So when you next hear of an access application that is integrated into the enterprise supply chain workflow, don’t say I didn’t warn you. Organizations who make ignorant decisions on the production-readiness of applications shouldn’t know why failures occur later, yet they do, and the engineers are left to pick up the pieces.

What Can You Do?

It’s not hopeless, really. It isn’t …. not necessarily fun, but there are strategies that you can attempt.

Gather

Create a personal archive of articles, use cases, scenarios, and data that allows you to tell stories to non-technical people, helping them understand the tradeoffs present in all organizations.

Internally, you might be interested in estimated vs. actual effort for feature delivery, production failure rates, or implementation costs mapped to the customer base. Are cloud costs increasing faster than customer growth? Did assumptions made during initial implementation impact the ability to deploy future features, whether positive or negative? Is supposedly important work upended by unknown and unplanned initiatives? Did a potential security breach impact customer confidence? What was the cost of researching a potential security breach?  Is data quality affecting your reporting, analytics, and billing? There are many different ways to try and understand what’s happening within your organization.

Almost daily, there are new articles online that highlight the issues and problems other organizations experience: Southwest’s 2022 holiday meltdown, a ransomware attack on Vital Care Providers, and Cloudfare’s bad software deployment. Not every organization publishes postmortems, but details often leak through other channels. Perhaps more importantly, your organization doesn’t want to appear in those articles!

Educate

As most non-technical folks appear unable or unwilling to accept that software is hard, our responsibility – for better or worse – is to show and explain. Unique situations require adjusting the story told, but it is necessary – and never-ending – to have any chance to get the organization to understand: explaining how software is developed and deployed, demonstrating how a data-driven organization requires quality data to make correct decisions, explaining the advantages and disadvantages of leveraging open source solutions; showing examples of how open source licenses impact your organization’s intellectual property. Look for opportunities to inject background and substance when appropriate, as education is open-ended and never-ending. Often, it will appear no one is listening as you repeat yourself, but eventually – hopefully – someone will parrot what you’ve been saying for months. 

Negotiate

Aside from those employed in purely research and development roles, engineering/technology for engineering/technology's sake is not feasible, as technology concerns must be balanced with business concerns: product and its competitors, sales pipeline, customer support and feature requests, security, privacy, compliance, etc. Each decision has its short- and long-term impacts, and it is very unlikely that all involved will be pleased. Sorry, but that’s corporate politics.

That does not mean you roll over and play dead, but rather horse trade, often with management and product, to ensure the technical concerns aren’t forgotten:

• Ensure that changes in business priorities are coupled with impact analysis on in-process development efforts;
• Accept less-than-optimal initial implementations with the agreement of fast-follow work to address compromises;
• Define metrics that identify when technology-focused work should be prioritized over feature work.

These ideas may or may not apply to your organization or situation, but hopefully, they will give you ideas that may be pursued. 

Conclusion

The problems I’ve discussed are age-old and have seemed to become worse in recent decades, so I’m not sure if any of what I’ve discussed is a surprise. Perhaps this is only the latest incarnation of the problem and post-Agile a new approach will reap benefits. Perhaps leaders will acknowledge that engineers really do understand the problems and are trusted to implement a solution rather than given solutions that fit an arbitrary (and often unrealistic) timeline. It’s a tug-of-war that I don’t yet see resolved.

Image Credits

• “Pointy Hair Boss” © Scott Adams
• “Productivity: Putting the Kanban Display Together” by orcmid is licensed under CC BY 2.0.
• “Analog circuit board prototype” by mightyohm is licensed under CC BY-SA 2.0.