How Homeland Security Went From One Deployment a Decade to “Moving at the Speed of Thought”

Mark Schwartz, who is the CIO at the US Department of Homeland Security, delivered a rare candid talk at the AWS government symposium in June. He discussed how outdated and manual process are killing innovation in the federal government, and what can be done to correct this.

He began by discussing, in a somewhat satiric style, the time it takes in government from when there is mission need, till a capability is deployed to meet that mission need.

“As soon as we realize we have a mission need, we wait,” said Schwartz. “We wait till we have a number of mission needs so we can assemble them and call it a program. The next step is to practice something we call “acquisition oversight”, what that means is that we write 105 long documents that nobody reads and we put them in a file cabinet. We do that because now that we have a big program with lots of requirements there is a lot of risk, so we have to exercise some oversight. Once we get the go ahead, we move at the blazing speed of procurement. We procure hardware and services to install the hardware in a data center. After GAO (Government Accountability Office) resolves (everything) the programmers start programing, then they start testing, then we go through a security accreditation process which takes a while and then we practice “change control” or “configuration control” to mitigate the risk that we might actually use the things we are creating. When you add all that up, the time to go from recognized need to capability is on the order of years to decades.

Mark believes this is unacceptable and has introduced a complete continuous delivery process to the Department of Homeland Security. The shift has revolutionized the department with one program that deploys multiple times per day others which deploy once or twice a week. The deployments run through a continuous integration pipeline and automated tests are performed so that if a vulnerability is discovered it can be patched without risk.

Mark says that innovation in government is easier than people think. “Government people have many great ideas but they don’t do anything about it because they are not empowered” says Schwartz. This is because they will have to write 105 pages of documents for each program. The way to create a culture of innovation is by minimizing the cost of an experiment through a continuous delivery systems. In the DOHS, if the team have a good idea, they try it because the continuous integration pipeline is there and all they have to do is put it in.

Schwartz concluded that “By reducing the costs and the risk of experiments, people will release all these creative ideas they have bottled away. I hear a new ideas every day from my team including how to do change control. All of these ideas were there just waiting to be released, we are now moving at the speed of thought”