Moving Towards Self-Adaptive Systems: ML-Based Auto-Tuning in Middleware

Introduction

A self-adaptive system has the ability to configure and reconfigure itself and continuously tune and optimize itself. Self-adaptive systems have been studied and solutions have been developed in a number of fields such as robotics, autonomic computing, control systems, distributed systems, and fault-tolerant computing. The main motivation for this work is to enable self-configuration into WSO2 products. Our products have various configuration parameters which we can tune to control the run-time behavior (performance, stability, etc.) of products. For example, the following table shows the (subset of) configuration parameters of the WSO2 stream processor.

The configuration parameters listed above have a direct impact on the performance of the Stream Processor. Although we have default values for these configuration parameters, these values may not guarantee the optimal operation (e.g. best performance) of the system (or the product) under all scenarios. The optimal configuration values will depend on many factors such as use case, traffic pattern, deployment pattern, hardware, and so on. Therefore, in order to achieve optimal operation, we may need to perform selective tuning. Such tuning may require significant effort in particular for large deployments that consists of multiple products. Furthermore, doing such specific optimizations may not always guarantee the optimal operation. The reason is that traffic patterns, use cases, hardware, etc. could change over time. When this happens, the system may no longer operate under its best-operating conditions. Auto-tuning will eliminate the need for finding optimal configurations and enable the system to adapt to changing conditions.

Primary Objective

Our primary objective is to come up with architectures and algorithms that can learn the behavior of a system on-line (i.e. on-the-fly) and self-adjust the parameter values to optimize certain objectives (or criteria). At a higher level, we can represent the architecture of the self-adaptive system using the following diagram using control theory.

As we can see, this model is similar to the generalized feedback control loop in industrial control systems. The feedback loop takes the system output into consideration, which enables the system to adjust its behavior to meet the desired objective. Here the objective can be a single objective (e.g. maximize throughput or minimize latency) or multi-objective (e.g. maximize throughput and minimize latency). One might wonder how it is possible to optimize both throughput and latency (at once) as the objective functions of throughput and latency are (generally) conflicting (i.e. as the throughput increases the latency also increases). In such cases, it is possible to find a set of suboptimal solutions (Pareto optimal solutions) and make decisions based on this set of suboptimal solutions (parameter values). We will discuss this in-depth in a separate blog.

In order to design self-adaptive systems, we need to first make the system observable. Once we make it observable we can monitor the system at run-time and collect the output metrics. For more information on how observability works, see Ballerina Observability Guide. Once we have the output metrics, we can feed these metric values into the controller. The controller will then decide on the optimal parameter values based on the conditions of the system. Designing the controller is a key part of designing a self-adaptive system. To design controllers we are using machine learning techniques (in particular Bayesian and reinforcement learning) as well as statistical-based methods.

Initial Results

Let us explore a recent project we did to autotune the number of workers in pipeline concurrency pattern. In this project, we developed a set of algorithms that can change the number of workers in the pipeline concurrency pattern online. The pipeline concurrency pattern consists of multiple stages where each stage consists of a queue (buffer) and a worker. There are several use cases we can implement using this pipeline concurrency pattern. For example, sentiment analysis where an application requires many data preprocessing stages such as sentiment classification and sentiment summarization. Furthermore, the pipeline architecture is extensively used in the image processing, 3D rendering, big data analytics and document classification domains. The self-adaptive model which we developed has the ability to change the number of workers, on-the-fly, based on the workload characteristics (i.e. input rate, the processing time of workers etc). The following figures show the performance comparison of adaptive and static models.

We note from the figures above that as the time progresses there is an increase in the throughput and degradation in the latency. This happens due to the increase in the arrival rate. The adaptive model detects the degradation in the performance and self-adjusts the number of workers to achieve better performance. We can see that both latency and throughput charts recover with the adaptive model.

The adaptive model outperforms (high throughput, lower latency) the static-model. We also have some interesting results for the thread pool concurrency pattern which is the most commonly used concurrency model in server-based systems. 

Conclusion

In this article, we discussed the reasons why we need to focus on designing self-adaptive systems. We demonstrated how self-tuning can improve the performance using one of the models we built. Our ultimate goal is to make all our products self-tune themselves. WSO2 research team conducts research related to distributed systems, performance, machine learning, block-chain, natural language processing, and stream processing.