Application Telemetry: Different Objectives for Developers and Product Managers

In the world of software development, making data-driven decisions is crucial. However, the approaches to data collection are often ambiguous, leading to misunderstandings between technical developers and product management teams. While developers focus on application performance and outage management, product managers are more concerned with user interactions and identifying friction points. Therefore, a comprehensive application monitoring strategy is often an afterthought in the development process. To avoid this pitfall, it is essential for development teams to build a shared understanding of telemetry objectives. A coherent team, aligned in their goals, can effectively measure and analyze data to drive meaningful insights. This article explores various data collection approaches for application telemetry, emphasizing its significance for both developers and product managers.

What Is Application Telemetry?

Application telemetry involves the automatic recording and transmission of data from multiple sources to an IT system for monitoring and analysis. It provides actionable insights by answering key questions for both developers and product managers.

For Developers

• How fast is the application responding to users in different geographic locations?
• Can the servers handle peak loads, such as during Black Friday?
• What infrastructure upgrades are needed next?
• How does the end-of-life of technologies impact application stability?
• What is the overall reliability of the application?

For Product Managers

• What are users primarily looking for?
• Should the company invest in building support for the particular operating system, browser, or form factor?
• What features should be developed or retired next?

Application Telemetry Objectives

It is important to note that both developers and product managers have different objectives and expectations from telemetry. Having a particular toolset implemented doesn’t cover it all. There are several monitoring tools available in the market, but one size doesn’t fit all. It is alright to have multiple data collection tools added to the application depending on the data collection goal. 

For Product Managers

• Product monitoring/feature usage: Understanding which features are most used and identifying areas for improvement.
• Backlog prioritization: Prioritizing development efforts based on user data and feedback.

For Developers

• Predicting future anomalies: Identifying patterns that may indicate future issues.
• Build failures: Monitoring build processes to detect and resolve failures quickly.
• Outage monitoring: Ensuring the application is available and performant.
• Security: Detecting and mitigating security vulnerabilities.

Techniques for Application Telemetry

Effective application telemetry involves a combination of tools and techniques tailored to the needs of both product managers and developers.

Here are detailed descriptions of key techniques and examples of tools used for each:

Product Manager-Focused Techniques

1. Application analytics: Application analytics tools collect data on how users interact with the application, providing insights into user behavior, feature usage, and overall engagement. These tools help product managers understand which features are most popular, which parts of the application might be causing user frustration, and how users are navigating through the application. 
   • Google Analytics: Widely used for tracking user interactions and engagement. It provides detailed reports on user demographics, behavior flow, and conversion rates.
   • Adobe Analytics: Offers advanced capabilities for real-time analytics and segmentation. It allows product managers to create detailed reports and visualizations of user data.
   • Mixpanel: Focuses on user behavior analytics, providing insights into how users interact with specific features and tracking events to measure user engagement and retention.
2. Real User Monitoring (RUM): RUM tools monitor the actual experiences of users as they interact with the application in real time. These tools collect data on page load times, user interactions, and performance metrics from the user’s perspective, helping product managers and developers understand the real-world performance of the application.
   • New Relic Browser: Provides real-time insights into user experiences by monitoring page load times, JavaScript errors, and browser performance metrics.
   • Dynatrace RUM: Offers detailed user session analysis, capturing every user interaction and providing insights into application performance from the user's perspective.
   • Pingdom: Monitors website performance and user experience by tracking page load times, uptime, and user interactions.

Developer Focused Techniques

1. Error tracking: Error tracking tools help developers detect, diagnose, and resolve errors and exceptions in real-time. These tools provide detailed error reports, including stack traces, environment details, and user context, enabling developers to quickly identify and fix issues.
   • Sentry: Provides real-time error tracking and performance monitoring. It captures exceptions, performance issues, and release health, allowing developers to trace problems back to their source.
   • Rollbar: Offers real-time error monitoring and debugging tools. It provides detailed error reports and integrates with various development workflows.
   • Airbrake: Monitors errors and exceptions, providing detailed error reports and notifications. It integrates with various development tools and platforms.
   • Elastic Stack (Elasticsearch, Logstash, Kibana): Elasticsearch stores and searches large volumes of log data, Logstash collects and processes logs, and Kibana visualizes the data, making it easier to identify and resolve errors.

2. Server Metrics: Server metrics tools monitor the performance and health of server infrastructure. These tools track metrics such as CPU usage, memory consumption, disk I/O, and network latency, helping developers ensure that the server environment can support the application’s demands.

• Prometheus: An open-source monitoring system that collects and stores metrics as time series data. It provides powerful querying capabilities and integrates well with Grafana for visualization.
• Grafana: A visualization and analytics platform that works with Prometheus and other data sources. It allows developers to create interactive and customizable dashboards to monitor server metrics.
• Datadog: Provides comprehensive monitoring and analytics for servers, databases, applications, and other infrastructure. It offers real-time monitoring, alerting, and visualization of server metrics.
• Dynatrace: Provides full-stack monitoring and analytics, offering deep insights into application and infrastructure performance, including server metrics.

The Phases of Application Telemetry

Application telemetry is a continuous journey rather than a one-time task. It should evolve alongside the application and its users. Implementing telemetry early in the development process is crucial to avoid compromising application stability and performance. Additionally, early integration provides valuable trend and timeline data for benchmarking.

1. Fundamental Step

• Track standard elements of application performance, such as loading times for most users.
• Collect descriptive statistics on users and usage patterns.

2. Business Insights

• Analyze user engagement and determine if users are interacting with the application as expected.

3. Prediction

• Predict trends and application failures using machine learning models.
• Analyze data to determine user needs and anticipate future requirements.

Early in the development cycle, a team can start by implementing basic telemetry tools like Google Analytics and Prometheus to monitor user visits, page load times, and server response rates, ensuring the app functions smoothly for most users. Don’t limit yourself by thinking that early on the application won’t have significant traffic. It is always beneficial to have these tools added early so that application telemetry forms part of the application’s core DNA. As the app gains traction, you can integrate Mixpanel and Dynatrace RUM to analyze user interactions with features like posting updates and commenting, identifying which features are most engaging and where users encounter difficulties. In this stage, ensure to include real-time user feedback collection, to have both quantitative and qualitative data collection. In the final mature stage, leverage tools such as Elastic Stack and Datadog to predict trends and potential issues, using machine learning to anticipate peak usage times and proactively address performance bottlenecks and potential failures, thus maintaining a seamless user experience and preparing for future growth.

Conclusion

Early implementation of an application telemetry strategy is vital for successful application development. By embedding telemetry tools at the onset, organizations can ensure that data collection is integral to the application’s core DNA. This proactive approach enables both product and development teams to make informed, data-driven decisions that benefit the application and its users. Application telemetry bridges the gap between these teams, allowing developers to monitor performance and predict anomalies while product managers gain insights into user behavior and feature usage. As the application evolves, leveraging advanced tools like Elastic Stack and Datadog ensures that trends are anticipated and issues are addressed before they impact the user experience. Ultimately, a robust telemetry strategy ensures that customer interests remain at the forefront of development efforts, creating more reliable, performant, and user-centric applications. Teams should prioritize telemetry from the beginning to avoid compromising application stability and harness the full potential of data-driven insights, benefiting both developers and product managers alike.