Adobe Service Runtime: Keep Calm and Shift Down!

Microservices at Adobe

Adobe’s transformation from desktop applications to cloud offerings triggered an explosion of microservices. Be it Acrobat, Photoshop, or Adobe Experience Cloud, they are all powered by suites of microservices mainly written in Java.

With so many microservices created, every developer had to go through the same painful processes, i.e., security, compliance, scalability, resiliency, etc., to create a production-grade microservice. That was the genesis of Adobe Service Runtime.

What Is ASR?

ASR or Adobe Service Runtime is an implementation of the Microservice Chassis Pattern. More than 80% of Adobe’s microservices use ASR foundational libraries. It offers cross-cutting concerns that every production-grade microservice is expected to have.

Highlights of the cross-cutting concerns included in ASR libraries:

1. Foundational libraries for Java and Python
   1. These libraries offer log masking, request stitching, breadcrumb trail, exception handling, async invocation, resiliency, etc.
   2. À la carte libs
   3. ASR connector libs/SDKs to talk to internal Adobe services
2. Blessed base containers are the security-blessed containers that accelerate container adoption for applications in any language.
3. Bootstrapping code and infrastructure templates for fast-tracking getting started.
4. Opinionated build system — i.e., how to build a Java application, run tests, launch debug setups, and package into containers.
5. Secure defaults for configurables to ensure teams get started with baselines that have been tested to work.

Having cross-cutting concerns as a single chassis helps organizations produce production-grade microservices at scale, just like an automobile manufacturer’s production line.

Why ASR?

Large organizations often have heavy compliance, security, resiliency, and scalability requirements. ASR provides a collection of foundational libraries, components, tools, and best practices (12 factor). This enables rapid development of four 9s-capable, innovative, and secure services. It also enables a container-first deployment system.

Value Proposition

We did a study on internal teams to derive the value proposition of ASR.

Using ASR saved Developers time and improved security posture by not reinventing the wheel.

Benchmarks

RPS

We did some benchmarking to see if ASR has any overhead over vanilla applications. We ran a ten-minute Gatling script to simulate 500 users, for example.

ASR Filters

Some cross-cutting concerns are offered as Filters, which can add some overhead. 

Our baseline comparison is the mean requests/sec of 20474.225.

Sections below show the performance change with individual filters disabled.

• ASR logging filter
  • The cost of disabling this is that the ASR service won't log incoming requests and outgoing responses 
  • Performance: mean requests/sec 21260.143, a 3.8% improvement
• ASR exception filter
  • The cost of disabling this is that stack traces can escape in exceptions, an ASSET violation
  • Performance: Mean requests/sec 20988.308, a 2.5% improvement
• ASR request ID filter
  • The cost of disabling this is that the ASR service won't have a unique request ID per request for tracking.
  • Performance: mean requests/sec 21354.042, a 4.3% improvement
• ASR request response filter
  • The cost of disabling this is that the ASR service won't automatically validate the Authorization header in the incoming request (if com.adobe.asr.enable-authorization-header-validation is set to true)
  • Performance: mean requests/sec 20896.923, a 2% improvement

The benchmarks reveal that using ASR adds minimal overhead when compared to the functionalities it offers.

Security 

CVE scans often uncover millions of vulnerabilities across codebases in large organizations. If Adobe developers had to manually fix each one, they would spend most of their time on patching rather than building features. By providing secure defaults and hardened components, ASR serves as a foundational library that reduces vulnerability exposure and saves developers valuable time.

CVEs

The Log4J incident is a testament to the success of ASR. When the CVE was published, users of ASR had to upgrade to just one version of ASR. Non-ASR repos had to scramble to migrate their libs off of Log4j. This clearly demonstrated the significant multiplier impact ASR has created within the company.

Sensitive Data in Logs

Log masking is another popular feature that is often recreated across the orgs. ASR comes with a modular log masking library that masks sensitive information. Logs that contain credit card, SSN, or any Adobe-defined sensitive info by default are automatically masked. Developers can also extend it to customize masking for additional use cases. This ensures consistent protection of PII across all applications.

ASR Connectors and Resiliency

ASR has connectors, which can be used to consume APIs exposed by other services inside Adobe. ASR connectors are application environment aware, i.e, a connector will automatically pick the right root URL of the service based on the app environment. For example, if the Application is running in the stage environment, the identity connector will use the identity stage URL; when the application is running in the prod environment, the identity connector will use the prod URL. This is possible due to the AutoConfiguration that ASR provides for all the connectors.

One of the challenges with microservices is that different SLAs are honored by services. Your service might have a higher standard, and you must often tolerate other services. By using ASR connectors, microservices get fault-tolerant communication out of the box. ASR connectors leverage Resilience4j to achieve this. Every connector comes with Resiliency features like bulkhead threadpool, circuit breakers, retries, exponential backoff, etc. 

By using ASR connectors, the posture of a microservice is greatly enhanced. There are guardrails in the thread pool that ensure there is no avalanche of threads. By using retries by default, the stress on Adobe's network is greatly reduced when the availability of the dependent service is degraded. This is a classic example of how pushing the cross-cutting concerns down to a common layer unlocks a lot of value and reduces redundancies.

ASR Adoption at Adobe

Almost every Java service at Adobe uses at least one of ASR’s libraries. The full suite of ASR is used by 80% or roughly 7000+ services at Adobe and continues to grow. With the growing need to make products more agentic, we see a strong need for libraries that support such use cases. ASR can be a powerful multiplier in enabling harm and bias guardrails, which are highly relevant to both the company and the industry today.

Keep Calm and Shift Down!

Inspired by shift left, shift down is a paradigm in platform engineering. A lot of cross-cutting concerns must be managed and provided by the platform out of the box. The users of the platform can focus on their functionalities without having to worry about the baseline standards set by Adobe. ASR enables shift down philosophy at scale.

Security teams and executives keep calm due to the centralization of best practices and the at-scale adoption of ASR. Developers are at ease due to overhead being handled at the foundational layer. 

Every company interested in developer productivity and operational excellence should adopt a shift-down strategy like ASR. Over the years, the ROI keeps compounding and helps companies move fast on paved roads that power the developer journeys.