Proactive Security in Distributed Systems: A Developer’s Approach

Once the product becomes famous and the customer base increases, it is no longer viable to serve the customers using simple systems without too many bottlenecks. Distributed software systems are inevitable, and it is directly related to the growth of the product. Building distributed systems is hard, and at the same time, it is fun. It is solving one big never-ending puzzle that gives you one unique challenge every day or every week depending on the size of the system. You cannot imagine any systems being built without security under consideration since for most organizations security is "job zero" or "job one," whichever order they follow. Now building the security into distributed systems being a big challenge is an understatement.

Software developers previously primarily focused on getting the product to work and tried to fit the security into the product later. This approach came with a lot of overhead cost and reworks. It has changed a lot in recent times since organizations started to focus on it more than before. Developers started considering security in the early stage of the project. However, this kind of adoption varies across various sizes of the organization and their maturity. This article is going to talk about a few common distributed security trends from recent security reports and how developers can try to defend them when building.

Security and Its cost in the Distributed Systems

With security becoming a priority, let’s look at its impact on distributed systems. If you are a mid to large scale distributed system, at that point you are already serving a bunch of customers, resulting in increased workload on your systems. Also, any impact to your system is already going to be big and affect hundreds and thousands of customers, if not more. It’s a huge responsibility. Any changes to this system should be carefully built, tested, released, and monitored. As much as the software side of things is getting important or more complex, the security side of things is equally getting complicated. Building a large-scale distributed system is expensive, and I am not going to get into it here. But secure distributed systems, or lack thereof, can get extremely expensive. If you consider security as an afterthought, it will end up being more work, and it may cause more damage. Because security costs are not just financial, and it can cause great harm to trust, reputation, and regulatory issues. Also, the aftermath of dealing with an incident in terms of operations is a cumbersome process. It involves almost every part of the company, including development, security, legal, PR, and executive. As research shows [1], the cost of the breach is much more expensive than fixing them early.

Why Known Security Issues Persist in Distributed Systems

Given these costs and continued defensive efforts, why do some security issues still not go away? There are multiple theories and practices out there on how to build safer distributed systems. This is not a new area. However, in the security space, even if the issue is well-known or identified, it takes really a long time to get fully fixed by all of the software out there. For example, the OWASP Top-10 web application threats have not changed much in the last 10 years, even though everyone knows this exists [2]. It is mainly because of multiple reasons, and to name a few: 

1. Software always changes, and changes are prone to error if it is not done correctly.
2. Attackers get innovative and find workarounds or other options to exploit the same attack vector. 

Distributed system security is not very much different; there are known issues that continue to exist for many years.

Lessons from the DBIR: Persistent Attacks in Distributed Systems

To understand these persistent issues, I will use the Data Breach Investigation Report (DBIR) [3] by Verizon Business, one of the reputed security reports in the industry, for this analysis. The report shows that some of the attacks have not changed in the last 10 years; for example, Distributed Denial of Service attacks (DDoS), credential-related attacks and web applications are one of the top attack vectors for breaches. So in this article, we will dive deep into some of the techniques using which developers can prevent the DDoS and credential-related attacks and how to scale this approach for other distributed system security issues.

Distributed Denial of Service Attacks

Based on the Verizon DBIR, denial of service attacks are continuing to be the top concerns over the past decade. As Fig. 1 indicates, the attack patterns are trending up as the web application usage grows.

Here are some of the steps developers can take to prevent it. This is not a comprehensive list but these are some critical ones that developers can control. Due to the article length, I am abstracting some of the concepts here and linking the references for future reading.

• Rate limiting: As a developer, the most critical step you could take to prevent this DDoS from happening is to enable rate limiting on your services. There are various levels of rate limiting you could consider: API-level limits and user-level limits. There are various rate-limiting mechanisms and implementations available for further reading [4].
• Monitor the incoming traffic: Often, a change in pattern can indicate whether someone is talking to our services more frequently, helping us decide how to respond. Be cautious with static volume limits—if the volume grows progressively, the focus should be on the rate of increase. 
• Load Shedding: This is another traffic congestion control mechanism which would be useful [5]. 

Credential Based Attacks

DBIR report shows credential-related attacks are one of the top issues for the past 10 years. It also makes sense that credentials are one of the critical factors when, once you get those, it gives the keys to the kingdom, so the attackers would go for it for a better return on investment for their efforts.

As I mentioned in the DDoS attacks section, there are various extensive tools and approaches the security organization may take. But here are some critical steps developers can take in their programs to reduce the surface area of these attacks:

• Use temporary credentials: Long term credentials are not your friend. They come with too many security risks and go for ephemeral credentials, instead.
• Log carefully: Logging is essential for debugging as well as investigating any operational and security issues in the future. But it is also vulnerable to log some sensitive data like credentials. 
• Multi Factor Authentication: This is one of the critical controls to prevent unauthorized access based issues. MFA should be enabled wherever possible. 
• Monitoring: Enabling credentials usage related monitoring is also another great way to prevent credential protection. The monitoring could be based on the user access pattern, locale and other parameters.

There is also a growing push to use passkeys instead of passwords, and a lot of work is happening in that space, which I won’t cover in this article.

Coding Security Into Distributed Systems

Now, we talked about how to tackle a couple of distributed system security issues. You may ask, “Don’t give me the fish, teach me how to fish?” That’s a fair point. Although some of the distributed system issues are common, it could very well vary depending on the industry and the region where your service operates in [6]. So, identify the common trend and also try to localize the issues specific to your industry, region. The best way to scope down the issues is to talk to your security teams and identify what issues might be more relevant to your software. Once identified, you can use the MITRE attack framework [7] as a reference; it will give various techniques attackers may use to break into your application, it will also recommend some solutions. This is a great way to think ahead and embed security during software development.

Conclusion

Building, maintaining, and operating distributed systems are hard, and doing that securely is even harder. In this article, we talked about how some of the distributed system security issues are not unique every year; rather, it is like “NullPointerException” in programming—it’s well known, but it happens often. However, the techniques and tactics used to do that are different, as the attackers get innovative as well. To prevent such issues, developers can take some common controls and, to scale ourselves, use the global data and apply the local context to it, such as the industry you operate in and the region you are part of. Besides making use of the local security teams, teams can use attack frameworks such as MITRE to get ahead of some of the security issues in the distributed systems. Developers can start today by applying these steps to their current or upcoming projects to build security in from the ground up.

References

1. HackerOne. (n.d.). Cost savings of fixing security flaws during development. Retrieved from https://www.hackerone.com/blog/cost-savings-fixing-security-flaws
2. Phoenix Security. (n.d.). OWASP Top 10 across the years: What are the exploited vulnerabilities? Retrieved from https://phoenix.security/owasp-top-10-across-the-years-what-are-the-exploited-vulnerabilities/
3. Verizon. (2024). 2024 Data Breach Investigations Report. Retrieved from https://www.verizon.com/business/resources/reports/2024-dbir-data-breach-investigations-report.pdf
4. Microsoft. (n.d.). Service protection API limits. Retrieved from https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/service-protection-api-limits
5. Amazon Web Services. (n.d.). Using load shedding to avoid overload. Retrieved from https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/
6. CrowdStrike. (n.d.). 2025 Global Threat Report. Retrieved from https://www.crowdstrike.com/en-us/global-threat-report/
7. MITRE. (n.d.). ATT&CK Framework. Retrieved from https://attack.mitre.org/

The opinions expressed in this article are solely those of the authors and not their employers.