Deployment Monitoring Tools — Kiali

Problem Statement…

In Microservices architecture, COE (Container orchestration cluster) will support to run thousands of container. How do we see the Canary deployment strategy progress on graphicle dashboard in each container’s logs, traffic distribution, protocols, service endpoints, service health check, application behavior and distributed tracing.

Each company may or may not be using different type of own automation scripts/tools to monitor these areas and also very difficult to quickly find the issue type in production environment.

If we have any common tools with graphical dashboard to managing the metrics and traffic routing, We can quickly find the issue and resolve it without taking much time.

Solution

There is a solution in market to solve all problem in one tool, called as “KIALI” to run this tool, should be available below environment to access KIALI graphical dashboard.

• Kubernetes or Open shift.
• Istio service mesh installed on top of Kubernetes environment to enable Kiali and dependent application as Prometheus.

With this environment, will do the Canary application deployment and monitoring traffic distribution, logs, protocols types, service endpoints, health check, application behavior and distributed tracing.

KIALI

“KIALI” is Management console for Istio based service mesh.

It provides dashboards, observability and lets you to operate your mesh with robust configuration and validation capabilities. It shows the structure of your service mesh by inferring traffic topology and displays the health of your mesh. Kiali provides detailed metrics, powerful validation, Grafana access, and strong integration for distributed tracing with Jaeger.

Canary Deployment

Canary release is a technique that is used to reduce the risk of introducing a new software version in production by gradually rolling out the change to a small subgroup of users, before rolling it out to the entire platform/infrastructure and making it available to everybody.

Kiali Architecture

Kiali Components

Kiali has composed of two components:

Kiali application (back end): This component runs in the container application platform and communicates with the service mesh components, retrieves and processes data, and exposes this data to the console. The Kiali application does not need storage. When deploying the application to a cluster, configurations are set in ConfigMaps and secrets.

Kiali console (front end): The Kiali console is a web application. The Kiali application serves the Kiali console, which then queries the back end for data in order to present it to the user.

Kiali Depends on External Services and Components…

Istio: Istio is the component that provides and controls the service mesh. Kiali depends on Istio and will not work if it is not present. Kiali needs to retrieve Istio data and configurations, which are exposed through Prometheus and the cluster API.

Prometheus: When Istio telemetry is enabled, metrics data is stored in Prometheus. Kiali uses this Prometheus data to determine the mesh topology, display metrics, calculate health, show possible problems, and so on. Kiali communicates directly with Prometheus and assumes the data schema used by Istio Telemetery. Prometheus is an Istio dependency and a hard dependency for Kiali, and many of Kiali’s features will not work without Prometheus.

Cluster API: Kiali uses the API of the Kubernetes Cluster (cluster API) in order to fetch and resolve service mesh configurations. Kiali queries the cluster API to retrieve, for example, definitions for namespaces, services, deployments, pods, and other entities. Kiali also makes queries to resolve relationships between the different cluster entities. The cluster API is also queried to retrieve Istio configurations like virtual services, destination rules, route rules, gateways, quotas, and so on.

Jaeger: Jaeger is optional but is installed by default as part of the Service Mesh installation. The Kiali console includes a tab to display Jaeger’s tracing data. Note that tracing data will not be available if you disable Istio’s distributed tracing feature. Also note that user must have access to the namespace where the control plane is installed in order to view Jaeger data.

Grafana: Grafana is optional but is installed by default as part of the Service Mesh installation. When available, the metrics pages of Kiali display links to direct the user to the same metric in Grafana. Note that user must have access to the namespace where the control plane is installed in order to view links to the Grafana dashboard and view Grafana data.

Why Our Solution Is Distinct…

Once you have several services deployed inside the service mesh, you will have several questions around Istio observability:

• How are the microservices connected inside the service mesh?
•  How much traffic is going to each service inside the service mesh?
•  How are the microservices performing (response time and requests per second, etc.)?
•  Are any of the microservices in an unhealthy state?
•  How can I troubleshoot issues with services running inside the service mesh?

Kiali provides the answers to the above questions and provides a big picture of the service mesh. The following features are amazing and there is no unique tool in market to get all these features.

• Topology
• Health
• Metrics
• Tracing
• Validations
• Wizards
• Configuration

Topology

The graph provides a powerful way to visualize the topology of your service mesh. It shows you which services communicate with each other and the traffic rates and latencies between them, which helps you visually identify problem areas and quickly pinpoint issues. Kiali provides graphs that show a high-level view of service interactions, a low level view of workloads, or a logical view of applications.The graph also shows which services are configured with virtual services and circuit breakers. It identifies security issues by identifying traffic not configured as expected. You can observe the traffic flow between components by watching the animation or viewing the metrics.HealthColors in the graph represent the health of your service mesh. A node colored red or orange might need attention. The color of an edge between components represents the health of the requests between those components. The node shape indicates the type of component such as services, workloads, or apps. The health of nodes and edges is refreshed automatically based on the user’s preference.

Metrics

Each detail view provides predefined metric dashboards. The metric dashboards provided are tailored to the relevant application, workload or service level.

Application and workload detail views show request and response metrics such as volume, duration, size, or tcp traffic. The traffic can also be viewed for either inbound or outbound traffic.

The service detail view shows request and response metrics for inbound traffic.

Validations

Kiali performs a set of validations to the most common Istio Objects such as Destination Rules, Service Entries, and Virtual Services. Those validations are done in addition to the existing ones performed by Istio’s Galley component. Most validations are done inside a single namespace only.

Galley validations are mostly syntactic validations based on the object syntax analysis of Istio objects while Kiali validations are mostly semantic validations between different Istio objects. Kiali validations are based on the runtime status of your service mesh, Galley validations are static ones and doesn’t take into account what is configured in the mesh.

Wizards

Kiali provides different actions to create, update and delete Istio configuration driven by wizards. These are located in the Actions menu on the Service Details page.

Some of the important wizards as Weighted Routing Wizard, Matching Routing Wizard, Suspend Traffic Wizard.

Configuration

The Istio configuration view provides advanced filtering and navigation for Istio configuration objects such as Virtual Services and Gateways. Kiali provides inline config edition and powerful semantic validation for Istio resources.

Risk (Vulnerability)

A vulnerability was found in Kiali allowing an attacker to bypass the authentication mechanism. Currently, Kiali has four authentication mechanisms: login, token, openshift and ldap. All are vulnerable.

The vulnerability lets an attacker build forged credentials and use them to gain unauthorized access to Kiali.

Additionally, it was found that Kiali credentials were not being validated properly. Depending on the authentication mechanism configured in Kiali, this could facilitate unauthorized access into Kiali with forged and/or invalid credentials.

Avoid Istio v1.5.0 for security reasons.