Here at Scalyr, we recently embarked on a full rewrite of our web client. Our application is a broad-spectrum monitoring and log analysis tool. Our home-grown log database executes most queries in tens of milliseconds, but each interaction required a page load, taking several seconds for the user.
A single-page application architecture promised to unlock the backend's blazing performance, so we began searching for an appropriate framework, and identified AngularJS as a promising candidate. Following the "fail fast" principle, we began with our toughest challenge, the log view.
This is a real acid test for an application framework. The user can click on any word to search for related log messages, so there may be thousands of clickable elements on the page, yet we want an instantaneous response for paging through the log. We were already prefetching the next page of log data, so the user interface update was the bottleneck. A straightforward AngularJS implementation of the log view took 1.2 seconds to advance to the next page, but with some careful optimizations, we were able to reduce that to 35 milliseconds. These optimizations proved to be useful in other parts of the application, and fit in well with the AngularJS philosophy, though we had to break few rules to implement them. In this article, we'll discuss the techniques we used.
An AngularJS Log Viewer
At heart, the Log View is simply a list of log messages. Each word is clickable, and so must be placed in its own DOM element. A simple implementation in AngularJS might look like this:
Second, each word had its own change watcher, which AngularJS would invoke on every mouse click. This was causing the lag on unrelated actions like the navigation dropdown.
Optimization #1: Cache DOM Elements
We created a variant of the ng-repeat directive. In our version, when the number of data elements is reduced, the excess DOM elements are hidden but not destroyed. If the number of elements later increases, we re-use these cached elements before creating new ones.
Optimization #2: Aggregate Watchers
All that time spent invoking change watchers was mostly wasted. In our application, the data associated with a particular word can never change unless the overall array of log messages changes. To address this, we created a directive that "hides" the change watchers of its children, allowing them to be invoked only when the value of a specified parent expression changes. With this change, we avoided invoking thousands of per-word change watchers on every mouse click or other minor event. (To accomplish this, we had to slightly break the AngularJS abstraction layer. We'll say a bit more about this in the conclusion.)
Optimization #3: Defer Element Creation
As noted, we create a separate DOM element for each word in the log. We could get the same visual appearance with a single DOM element per line; the extra elements are needed only for mouse interactivity. Therefore, we decided to defer the creation of per-word elements for a particular line until the mouse moves over that line.
To implement this, we create two versions of each line. One is a simple text element, showing the complete log message. The other is a placeholder which will eventually be populated with an element per word. The placeholder is initially hidden. When the mouse moves over that line, the placeholder is shown and the simple version is hidden. Showing the placeholder causes it to be populated with word elements, as described next.
Optimization #4: Bypass Watchers for Hidden Elements
We created one more directive, which prevents watchers from being executed for an element (or its children) when the element is hidden. This supports Optimization #1, eliminating any overhead for extra DOM nodes which have been hidden because we currently have more DOM nodes than data elements. It also supports Optimization #3, making it easy to defer the creation of per-word nodes until the tokenized version of the line is shown.
Here is what the code looks like with all these optimizations applied. Our custom directives are shown in bold.
sly-repeat is our variant of ng-repeat, which hides extra DOM elements rather than destroying them. sly-evaluate-only-when prevents inner change watchers from executing unless the "logLines" variable changes, indicating that the user has advanced to a new section of the log. And sly-prevent-evaluation-when-hidden prevents the inner repeat clause from executing until the mouse moves over this line and the div is displayed.
This shows the power of AngularJS for encapsulation and separation of concerns. We've applied some fairly sophisticated optimizations without much impact on the structure of the template. (This isn't the exact code we're using in production, but it captures all of the important elements.)
To evaluate performance, we added code to measure the time from a mouse click, until Angular's $digest cycle finishes (meaning that we are finished updating the DOM). The elapsed time is displayed in a widget on the side of the page. We measured performance of the "Next Page" button while viewing a Tomcat access log, using Chrome on a recent MacBook Pro. Here are the results (each number is the average of 10 trials):
The "data fetched from server" figures include time for an AJAX call to our backend to fetch the log data. This is unusual for the Next Page button, because we prefetch the next page of logs, but may be applicable for other UI interactions. But even here, the optimized version updates almost instantly.
This code has been in production for two months, and we're very happy with the results. You can see it in action at the Scalyr Logs demo site. After entering the demo, click the "Log View" link, and play with the Next/Prev buttons. It's so fast, you'll find it hard to believe you're seeing live data from a real server.
Implementing these optimizations in a clean manner was a fair amount of work. It would have been simpler to create a single custom directive that directly generated all of the HTML for the log view, bypassing ng-repeat. However, this would have been against the spirit of AngularJS, bearing a cost in code maintainability, testability, and other considerations. Since the log view was our test project for AngularJS, we wanted to verify that a clean solution was possible. Also, the new directives we created have already been used in other parts of our application.
We did our best to follow the Angular philosophy, but we did have to bend the AngularJS abstraction layer to implement some of these optimizations. We overrode the Scope's $watch method to intercept watcher registration and then had to do some careful manipulation of Scope's instance variables to control which watchers are evaluated during a $digest.
At Scalyr, we're all about improving the DevOps experience through better technology. If you've read this far, you should probably read more about the Scalyr log management tool.