Streamlining Incident Management with IBM Cloud Logs, Event Notifications, and PagerDuty

In today’s fast-paced cloud environments, efficient incident management is crucial for reducing downtime and improving the customer experience. In this article, we’ll walk through a practical use case where a fictional company, ABC Ltd., leverages IBM Cloud Logs and Event Notifications to streamline their incident alerts to PagerDuty, ensuring timely responses to critical events. We’ll also cover how to integrate notifications with Slack and Email for different team members.

Use case: Managing application logs in a hybrid cloud environment

ABC Ltd. hosts its web application across multiple cloud regions, ensuring high availability for its global customers. Monitoring logs for errors and performance issues in real time is essential to maintaining uptime. To automate incident responses, they want:

1. Error detection: Real-time alerts for critical application errors and performance issues.
2. Incident notification: Integration with PagerDuty for urgent incidents, Slack for team discussions, and email for stakeholders.
3. Reduced noise: Filtering irrelevant logs to avoid alert fatigue and focus only on high-priority issues.

Architecture overview

1. IBM Cloud Logs collects and centralizes logs from ABC’s application, servers, and network devices.
2. IBM Event Notifications routes the logs based on defined rules to various channels: PagerDuty, Slack, and Email.
3. PagerDuty receives real-time alerts for critical incidents, while less urgent alerts are sent to Slack and Email for further analysis and collaboration.

Step-by-step configuration

Step 1: Configure Event Notifications

1. Provision Event Notifications:

• From the IBM Cloud Catalog, search for Event Notifications, choose a plan best suited to your needs and click Create.
• Navigate to the Event Notifications service from the IBM Cloud dashboard.
• Select a region to host your notifications and click Create. This service acts as the intermediary between your log events and notification channels.

Step 2: Set up IBM Cloud Logs

1. Provision IBM Cloud Logs:

   • Log in to your IBM Cloud account.
   • From the Catalog, search for IBM Cloud Logs and click Create.
   • Configure your log collection to gather logs from your applications, infrastructure, and cloud services.

2. Log Source Setup:

   • Use FluentBit agents to forward logs from your environments to IBM Cloud Logs. For example, you can install a FluentBit agent on your virtual machines or Kubernetes clusters.

3. Customize Log Parsing:

   • You can use parsing rules to extract key metrics or error patterns that are important to monitor. IBM Cloud Logs supports parsing with regular expressions to capture specific data from log lines.

Step 3: Link IBM Cloud Logs to Event Notifications

1. Authorize Event Notifications:

   • Go back to your IBM Cloud Logs instance, Open Dashboard and navigate to Integrations > Outbound Integrations (sidebar).
   • Select Event Notifications>Add>Add New from the integration options.
   • Follow the prompts to authorize the connection. You will need to assign IAM roles that allow Event Notifications to receive logs from IBM Cloud Logs. (Under the IAM Authorisation, set Source as Cloud Logs and Target as Event Notifications).
2. Configure Filters for Logs:
   • Use filters to only send critical logs to Event Notifications. For example, you can set a filter to trigger notifications when a log contains the keyword "ERROR" or "CRITICAL".
   • This helps reduce noise by sending only high-priority logs to PagerDuty and less urgent ones to Slack or email.

Step 4: Set up alerts 

1. Configuring Alerts:

• Navigate to your IBM Cloud Logs Instance and click on the Menu icon>Alerts>Alert Management>New Alert.
• Set the name, description and the severity of the alert you want to see. For example,

• Next, add the query. 

• Set the frequency of the alerts and how you want to be alerted.

• Once you have set all the parameters, click create alert.

Step 5: Setting up destinations, alerts, topics, and subscriptions

1. Create Notification Topics:

• In Event Notifications, create topics to categorize your alerts. For example, create topics like:
  • Critical_Alerts for PagerDuty.
  • Team_Discussions for Slack.
  • Status_Updates for Email.
  • Example:

2. Add Subscriptions:

• For each topic, add a subscription to direct the alerts to the appropriate channel:
  • PagerDuty: Select Webhook as the destination type. Follow PagerDuty's integration guide to create a webhook URL, and configure this as the destination for critical alerts.
  • Slack: Create a Webhook URL in your Slack workspace. In Event Notifications, select Webhook and enter the Slack URL for team notifications.
  • Email: Choose Email as the destination, and add the relevant addresses for stakeholders who need regular updates but are not directly responsible for immediate incident response.
  • Example: 

3. Set Up PagerDuty Alerts

1. Create an Alert Service in PagerDuty:

   • Log in to your PagerDuty account and create a new Service that will handle the IBM Cloud Logs notifications.
   • In the service, create an Integration for incoming webhooks and copy the URL provided by PagerDuty.

2. Connect Event Notifications to PagerDuty:

   • Return to IBM Cloud’s Event Notifications service, select your Critical_Alerts topic, and add the PagerDuty webhook as a destination.
   • Now, whenever an alert is triggered by critical logs, PagerDuty will receive the notification and escalate according to your team’s on-call schedule.

4. Set Up Slack Integration

1. Create a Slack Webhook:
   • In Slack, go to App Directory and search for Incoming Webhooks. Configure a new webhook URL in the Slack channel where you want notifications to appear.
2. Add Slack as a Destination in Event Notifications:
   • In IBM Cloud’s Event Notifications, add the Slack webhook URL to the Team_Discussions topic subscription.
   • This will send non-critical but relevant logs to Slack for team discussions.

5.  Configure Email Notifications

1. Set Up Email Destinations:
   • In Event Notifications, create an Email subscription for the Status_Updates topic.
   • Add the email addresses of team leads, product owners, or stakeholders who need periodic status reports on log activity.

Step 7: Test the Configuration

1. Trigger Test Logs:

   • Generate a test error or critical log event in your application.
   • Check that PagerDuty, Slack, and Email are receiving notifications as configured.

2. Monitor the Flow:

   • Confirm that the right alerts are sent to the right channels without causing unnecessary noise. Adjust your filtering criteria as needed.

Benefits of this setup

• Faster Incident Resolution: By integrating IBM Cloud Logs with PagerDuty, you ensure critical incidents are addressed immediately, reducing downtime.
• Team Collaboration: Slack integration helps your team discuss and resolve less urgent issues in real-time.
• Visibility for Stakeholders: Email notifications provide regular updates for stakeholders without overwhelming them with every minor issue.
• Customizable Filters: IBM Cloud’s event filtering ensures that only the most relevant logs are sent to each destination, reducing alert fatigue and ensuring the team stays focused on important tasks.

By integrating IBM Cloud Logs with Event Notifications and popular tools like PagerDuty, Slack, and Email, ABC Ltd. improves incident management, streamlining the process for both technical and non-technical team members. This setup helps them provide a superior customer experience with faster issue resolution and more efficient collaboration.