Can We Trust Our Software?

DZone 's Guide to

Can We Trust Our Software?

Software developers may have an understanding about their software's security, but what about everyone else?

· Agile Zone ·
Free Resource

Jeff and I question how trustworthy the software that has infiltrated every aspect of our lives is. To help us think about that we invited Dr. Bill Curtis, Director for the Consortium for IT Software Quality (CISQ) to join the conversation. He provides some insight into why CISQ has recently launched the Trustworthy Systems Manifesto. You can listen to the Podcast here.

Jeff Fraleigh: My name is Jeff Fraleigh and with me, as always, is my wingman, Pete Pizzutillo. And today we welcome Dr. Bill Curtis. Dr. Bill Curtis is an industry luminary. We were just counting the number of decades that he has been out there helping people build better software. All the way back to TerraQuest and Borland, SEI. Bill's joining us today. He's got a lot of interest and passion throughout the years. He's currently working on something really interesting, and it's framing up the conversation in this episode. We start our statement of the day, and today the statement's is: Can We Trust Our Software?

Pete Pizzutillo: It's not a question only we are asking; consumers and business leaders are out there are asking daily. And trust is an interesting concept that hasn't been joined with software. Bill has led a group that has published this manifesto: The Trustworthiness Systems Manifesto.

Jeff Fraleigh: So why a "manifesto?" Let's back up a little bit. It's a great term, it sounds like a dish you can get your local Italian joint but what is it?

Why Create the Trustworthiness Systems Manifesto?

Dr. Bill Curtis: You already got some manifestos out there such as The Agile Manifesto and Rugged Software Manifesto. These are written from the point of view of programmers. And, they don't really deal with whether you're delivering trustworthy software to the business, nor whether the business can count on it. The Rugged Manifesto does that a little bit, but again, it's still from the point-of-view from the developer. And the truth of the matter is, no developer ever got whacked because he delivered bad software. The people that own the risk are the executives of the company, not the CIO — the people that CIO reports to.

When the system goes out, the Wall Street Journal doesn't call the CIO, they call the CEO, who doesn't know squat about software. So, we needed a manifesto for the senior leaders of the company, who own the risk and need to be able to communicate to IT or to engineer, what they expect. "This is what we expect in order for you to reduce the risk to the business of software-intensive systems." Because when these things go out, in the old days you lose five minutes or so, but now these outages cost over a hundred million dollars. We're in the era of the nine-digit glitch. And, that's not bits and bytes, that's dollars and euros.

For example, Knights Capital trading four-hundred-and-forty million dollars in bad trades in thirty minutes, and they went bankrupt. You look at all these British banks going down and airlines have outages that can keep planes on the ground for a day and a half, and that's way over a hundred million dollars in lost revenue. Bottom-line is we're looking at the systems that produce enormous risk for the business. Especially as more of our business functionality, and our product functionality, is put into software. The modernization of the business puts the whole business process on software, and it’s putting businesses at risk if the software can't be trusted.

The issue here is the trustworthiness of the software, and whether the business can feel confident, that it's minimized the risk to the business, to its shareholders, to its employees, to its customers. When it fields these large systems, either in products or in business services.

Jeff Fraleigh: That makes a lot of sense. Though, why choose to put this in manifesto form versus any other form? Is there a specific value in calling it a manifesto or writing it as a manifesto versus any other type of document, or white paper, or opinion paper?

Dr. Bill Curtis: Well, it tends to get a little more attention, number one. Number two, it's simply a statement of principles that we're dead serious about. That's what a manifesto is. It says, "I am serious about the following things." And we didn't put a lot into the Trustworthiness Systems Manifesto, we started with a lot of different principals then boiled it down to five. We looked at other definitions of trustworthiness. One that's being developed in the ISO and being used by NIST and the Industrial Internet Consortium. And found that we're pretty well matched, in fact very heavily matched, to what they're listing as the issues that are in trustworthy systems.

Dr. Bill Curtis: They've got basically five: it's reliable, it's resilient, it's safe, it's secure, and it maintains privacy. And we've included all those in our principles. We boil it down to five principles, and this is what we want executives to begin a conversation with IT or with product engineer, saying "Look, I wanna know how you are installing these as the way we work. That you reduce the risk to the business. Because we know that these are issues that create risk, and we want you to manage it. I want to know how you're going about doing this. I want you to let me know. Let's track it." And this has been done as part of governance. The Trustworthiness Systems Manifesto is written to how we ought to govern the risk to the business that the software-intensive system creates.

Jeff Fraleigh: It's interesting to know that executive management hasn't had something to point to in order to start or have a constructive conversation. You keep using this word "we," and I know that this is created by CISQ. Can you give an overview of what is CISQ?

About the Consortium for IT Software Quality

Dr. Bill Curtis: CISQ was started in 2010 by the Software Engineering Institute at Carnegie Melon, and the Object Management Group, which is an international standards organization primarily dealing in software-intensive areas, because they were getting requests from system integrators, who were finding requirements and contractual issues in their contracts. Basically, the equivalent of service level agreements, for things like reliability and security and maintainability. The problem is, every company had a different definition of what that meant. And they said look, "We can't deal with seventy-five different definitions of reliability or security or what-have-you. We need international standards we can all agree to and keep the damn lawyers out of it." Then we formed CISQ, they ask me to be the executive director since I've worked with both organizations. And the initial goal was to define automate-able measures. Automate-able being critical here. Automate-able measures of software size and quality. And we held executive forums in Washington D.C., Frankfurt, Germany, and Bangalore, India. And asked executives from the consortium, "What measures would you like to undertake initially?" And they came up with five measures. We voted and what not, and the five measures that germinated — that came out as the clear agenda:

Number one, a measure of size: they wanted to measure function points in an automated way as close as possible to the IFPUG- the International Function Point Users Group counting guidelines. Because they wanted to use function points, but they were getting fed up with the cost and inconsistency in different manual counts. They wanted to automate it make it consistent and cheap. So we've done that, it's in use, it's supported commercially, and we've submitted that standard over to ISO for their consideration for adoption.

The four quality measures they asked to automate were: reliability, security, performance efficiency, and maintainability. And we built those out of known severe weaknesses. We didn't take anything that wasn't severe. If you can defer it forever, we didn't include it in the measure. The measures really count the number of severe weaknesses that are detected through static analysis and a piece of software. And develop the measure from those. They really do represent the risk of a piece of software in those four areas. Three of them being operational, one of them being cost.

And those are now out there, they're in use, they're supported commercially. And we have just recently updated that, by looking after those that had been out there three or four years to see were there severe weaknesses we need to add to it, and we found a number. That- about twenty or thirty that we've now added into the measures. And that's now going through the OMG standardization process. So, when we complete these they go through the OMG process, which is very efficient, and when it's done it's an automate-able specification supported by OMG as a standard. And released for free. I mean, it's a free standard. And vendors like CAST and others can pick it up and automate in their technology.

Pete Pizzutillo: We've been watching CISQ grow for a while. So, why release the Trustworthy Systems Manifesto now? What is that you're seeing in the market?

Dr. Bill Curtis: I'm seeing nine-digit glitches in the market! And they're not slowing down. I mean if you get the hacker news, there's a new hack somewhere. It's huge, these things are not small, they're huge. And you're seeing British banks, they seem to go down once a week. The bottom line is, it is getting worse. And we're putting more and more of our business functionality, of our product functionality into the software. And the software is not getting better. Gartner's estimating it's getting worse. That the amount of technical debt being produced is staggering because Agile, while it gets software delivered faster, doesn't offer the same amount of opportunity for quality assurance that we used to have. And a lot of people aren't really accepting the discipline of Agile methods — they're hacking. And they use Agile as an excuse for that, and then we get a lot of complaints from executives that what they're saying is not a discipline performance of an Agile discipline. And the executive team is very concerned because they know the risk now is enormous. And they need some way to try to turn the corner on the trustworthiness of what they're betting the business on.

Pete Pizzutillo: I would add to that other than the expensive nature of these failures, the public visibility — I would that suggest that digital transformation, cloud, and DevOps can be a triangle of doom if you haven't gotten the foundation buttoned up. You're releasing software faster than ever before, at a scale greater than ever before to the end user. There's a lot of exposure.

Dr. Bill Curtis: The minute an airline system goes down there's ten thousand people in the terminal tweeting their anger to the entire world.

Pete Pizzutillo: There's a lot of concepts in the Trustworthy Systems Manifesto: trustworthiness and accountability. I would say what I notice differs in this manifesto than others, is like you place the business stakeholder and IT leadership is at the center of accountability. Not so much at the center of risk. We started this conversation saying that they're the ones that lose the job, but essentially you're trying to get them to stand up and say, "Yeah I will own this problem. I will be the person that makes sure that we get to this vision." Am I characterizing that correctly?

What are the Principles the Trustworthy Systems Manifesto Defines?

Dr. Bill Curtis: The goal is, because they're not IT experts, right? They didn't graduate in software engineering or IT, they're business people. And some of them do have engineering backgrounds, but the fact is they haven't been current in the processes, the methods, the languages, the technologies — the technology stack that represents most modern applications, and certainly represent the internet and things. And they need some guidance on "what do I communicate" to my folks in engineering or in IT to say, "This is what we've got to do in order to reduce the risk." And there are five areas that coalesced it all down to, we boiled it down to sort of five principles.

Number one is just obvious: engineering discipline in product and process. Because what we're hearing is a lack of discipline. That too many people are using some of the modern manifestos, the Agile approach, as a way to just do what they want to do. From the point-of-view of the programmer, they don't really accept fully the discipline of Agile or SAFe, or newer methods. And when you go into a DevOps environment, that's even more critical. Second: the business needs to say what level of risk they can tolerate. You can't get to zero defects. One, it's almost theoretically impossible. Getting even close to it is an excessive cost that most companies can't afford, the question is, "What level of risk can we tolerate, and then how do we tune our quality assurance methods to make sure we're operating within that risk tolerance?" but that takes time.To calibrate what you're doing, and how you're measuring it, and how you're testing it- to make sure it can operate within your tolerance.

Third: most modern apps are built from a long source, a lot of sources. It's really a supply chain. You've got open source components, things from third-parties and stuff you've built internally. Sometimes on-shore, sometimes off-shore. And the truth of the matter is you need to know where these components come from, what their vulnerabilities are if they're subject to license agreement you didn't know about. There's a lot of risk in all these components. You need to know what their properties are. It's traceable properties of system components. Then you will know what the providence of these things is. "What licenses might I be subject to. Are there known vulnerabilities that haven't been patched?" Look at what happened to Equifax. Right, so that becomes important.

Number four, and this is so obvious: proactive defense of the system and its data. But we're really talking about layered defense because modern systems have hardware and software — you've got your data, there are networks, all kinds of things — you need protection at all these different levels. And we even heard someone who was the head of cybersecurity at the Department of Homeland Security at a conference say, "Look, the only safe assumption you can make is they've already broken into your system — they're already inside. Now, how do you protect your data, once they're in?" Again, that's a different layer you have to worry about. A much more serious understanding of what it takes to provide adequate security of a system. And then finally: resilient and safe operations. Which is obvious, but again, that's not an issue you want to find out about once you have a problem. And that really becomes an issue of how you build software, how you design software, how you test it to make sure you achieve whatever your tolerance thresholds are. So that you can have resilient and safe operations.

Dr. Bill Curtis: Those are five principles. And that covers what we're hearing from ISO and the National Institute of Standards in Technology talk about as trustworthiness — as a definition of the issues in trustworthiness. We don't expect executive teams to understand all the nuances of these five principles, but it gives them guidelines of what to talk about and what to really expect from the organization. And what they want to hear back are plans and evidence that we're doing these things. That we are in fact implementing practices that will reduce the risk of software-intensive systems to the business.

How Do You Put the Trustworthy Systems Manifesto in Practice?

Jeff Fraleigh: That's incredibly interesting, but what am I going do if I want to do this tomorrow? What should my steps be to begin this process?

Dr. Bill Curtis: Number one: go to the CISQ website www.it-cisq.org and right there on the homepage you'll see the manifesto and can download it. It has descriptions of these principles, and things you can do. What you ought to expect, behind each principle. We encourage you to sign the manifesto, we're trying to get a lot of people involved, especially executives — get them to sign it saying, "I agree with this. I'll work on it in my organization. I'll communicate it to my people and to other organizations." Number one, read it and sign it, then take it home to your organization and start the conversation: what are we doing about these five areas to reduce the risk? If you're an executive, this needs to become an open conversation and purposeful conversation with your engineering department or your IT department.

Jeff Fraleigh: And you feel like that's the first thing that someone should do to be able to not only say "I read it, I get it," but to really commit to wanting to, and learn directly from the CISQ site.

Dr. Bill Curtis: Exactly, and then take it home. It really only matters if it's used in corporations to generate this discussion and have executives say "This is what I expect" and get their departments, IT or engineering, to respond with evidence of what they're doing to implement these practices.

Pete Pizzutillo: And I would add that there are local CISQ groups. I know you were just out in India, helping people think about this. I know there are folks in D.C. and other places across Europe. So, look at the resources that are available through CISQ to figure out when those meetings are and those working groups in the overall community.

Bill, thanks for joining us today. It's a tremendous endeavor that you're taking on here, hopefully, we can help you spread the word- but an important one. I'm glad that we have somebody with passion, depth to help us drive through this. You're right I think its nine-digit issues. There's a lot of turbulence in the marketplace. There's a lot of exciting things happening. So, I mean I think the way I look at this, I think that trust is such a great positive word. It's not looking at the negative pieces, the failures or the opportunities, and I think we're all counting on our software to be trustworthy. And that resonates up and down the food chain. And something that we all need if we're going get into the smart economy, and the digital economy, and all where our systems are going. If you think about Alexa and Facebook, all the stuff that's going on there where our personal lives are so infiltrated with the software. A lot of us haven't really thought through what the impact is to our families and the future of our families. But if we can start facing the problem now, I think it's a great way for us to be smart about how we move forward and how we innovate. Good luck with everything there. Hopefully, we'll have you back, we can get an update on who signed the manifesto.

application development ,software reliability ,standards body ,software engineering ,agile ,trustworthiness

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}