Over a million developers have joined DZone.

Search For Massive-Scale Log Analysis

DZone's Guide to

Search For Massive-Scale Log Analysis

Log management expressly for interactive response for searches on hundreds of terabytes of data or more.

· Performance Zone ·
Free Resource

Sensu is an open source monitoring event pipeline. Try it today.

Loggly, the company behind the most popular cloud-based, enterprise-class log management service, today announced Gamut™ Search, the first log analysis technology specifically designed to deliver near instant response for searches over massive data volumes covering long time periods. Rather than waiting hours or days for a query to complete, the new capabilities enable users to begin analyzing the entire range, or the “gamut” of their log data, immediately, in a highly interactive fashion.

“Consider that a single movie at TV quality is a gigabyte,” said Charlie Oppenheimer, CEO at Loggly. “Companies are now generating thousands of gigabytes of log data or more every day, but until now most log management platforms have bogged down when processing searches and analyses at this scale,” “Gamut Search represents a new architectural approach based on years of experience with billions of interactions and petabytes of data. There’s is no longer a need to constantly reduce scope. More importantly, these productivity gains translate into faster mean time to resolution and fewer revenue leaks.”

Gamut Search changes users’ relationship with log data. Previously, users would set their large queries, wait for them to run, and then receive the results. Gamut makes the entire log analysis process more interactive and iterative, allowing users to see instant results that can quickly be modified and optimized based on search terms. With Gamut Search, each search request is broken down into small time slices that are computed dynamically. The most recent time slice is processed first and the results of that time slice are presented instantly. The system then progresses through each earlier slice until the entire request has been fulfilled. With Gamut Search, you get results immediately, instead of waiting for the entire search to be processed before seeing results which means users can take action more quickly.

“Loggly Gamut Search really is a tremendous advance. Retrieving search results is much faster, and I can start browsing recent events almost instantly while older data continues to load in the background,” said Ryan Jung, software engineer at ClearCare. “The new UI gives me a clean visual reference for the loading process. The near instant access to data browsing is a crucial feature. I can get to the bottom of my problems right away and without endlessly spinning throbbers, which makes me a much happier engineer. Gamut Search is making log interaction easier by degrees.”

“Not having to wait for search results to become available makes a huge difference”, said Jessie Chen, software engineer at MedHelp.org. “Gamut Search allows for much more agile interaction with large data sets. Refining a complex search pattern is a lot easier when you can see its results right away. It simply allows you to be much more efficient. No wait time means less stress and better focus on solving the problem.”

“Getting to the root cause of the issues used to be a painful task that involved long and unproductive wait times“ said Mayank Singh, product manager at Indix. “And when you’re debugging a critical problem that is impacting your customers and revenue, you neither have the time nor patience. Gamut Search gives me results as they’re being found. It’s really fast and I can get to work immediately.”

With every search, users see a graphical event timeline that shows when the log events represented in the search results occurred, simplifying navigation over time periods. In addition, they have one-click access to the full unfiltered collection of events that occurred immediately before or after an event of interest via Loggly Surround Search.

With Gamut Search, results are also strategically cached to provide even faster response for subsequent requests. “As part of Loggly’s ongoing analysis of user behavior, we have discovered that more than 50 percent of all search and analysis interactions are based on subsets of previously queried data within the last 30 minutes,” said Jon Gifford, founder and chief search officer at Loggly. “For this reason, Gamut Search provides memory-speed response for subsequent analyses, further boosting interactivity.”

Summary Benefits

  • Faster mean time to resolution eliminates revenue leaks: Users get near instant search results across huge data volumes and over long time periods.
  • Time savings for developers and DevOps: Users benefit from the pre-processing that Loggly does on log data and can iterate on their queries much faster than before.
  • A more satisfying user experience: Users start seeing data immediately and have greater transparency on query response through progress indicators.
  • Usability improvements eliminate many of the annoying things about searching logs, such as re-generating queries multiple times to get a slightly different time slice.

Sensu: workflow automation for monitoring. Learn more—download the whitepaper.

log management ,log analysis ,performance

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}