Capturing AJAX API Requests From Arbitrary Sites With a Chrome Extension

I assumed there would be a lot of Chrome extensions to monitor HTTP requests from AJAX calls, but the few that we found (such as the Postman Intercept Chrome Extension) seem to only capture the request only and not the response. Even though Chrome DevTools has the network tab, it's hard to share those captured HTTP traces with teammates.

Our company, Moesif API Analytics, is very API-driven internally and externally. We divide responsibilities among API boundries, but we constantly have to copy and paste the HTTP headers and JSON payload from the DevTools into email or Slack when debugging API issues. Sending multi-megabyte *.har files through Slack is just as cumbersome. In addition to not being easy to share, inspecting HTTP traffic in Chrome DevTools is not ideal because there is no easy way to filter the API Calls or inspect JSON payloads. The network tab is designed for traditional HTML websites rather than modern single-page apps and APIs that return JSON.

Due to this time sink, we decided to build a Chrome Extension to make capturing and debugging these AJAX requests (and responses) from any website real easy. The extension is designed for REST APIs in mind, such as those powering single-page apps. However, some monkey business was needed to do the capturing that went around the Chrome Extension APIs due to limitations in Chrome’s APIs.

Issues With WebRequest Approach

Chrome WebRequest API is part of the Chrome Extension set of APIs.

Initially, I looked at the Chrome’s WebRequest APIs to capture API calls from an open browser tab. We leveraged those APIs before for our CORS and Origin Changer. It seemed to be the most natural place to capture the data.

However, we ran into a bug where the WebRequest API doesn’t expose an interface to read the response body. See the issue here on Chromium.

While we were surprised by the lack of a response body getters, we now understand a possible reason why most other extensions do not capture the responses. Our requirement was to capture the response so that they could be shared with team members. Without responses, debug would be painful and resolving issues would be slow.

XmlHttpRequest Event Listeners Approach

Since XmlHttpRequest is used by all browsers for AJAX calls, I leveraged this to monitor the API calls through monkey patching.

There are several XmlHttpRequest listeners for hooking into:

• loadstart.
• progress.
• abort.
• error.
• load.
• timeout.
• loadend.
• readystatechange.

From the listener callbacks, I can get the event target, which is the XmlHttpRequest object itself. Looking at the XmlHttpRequest api documentation, I realized it has exact the opposite problem as WebRequest API. The object has interfaces to get the HTTP response but not the original request. There is a setRequestHeaders(), but no getter method to pair with it to get the request HTTP headers. In fact, it doesn’t even have a public method or property to get the original URL or path.

XmlHttpRequest Monkey Patch

To truly monitor all the data (i.e. request headers/body and response headers/body), I’d have to monkey patch it. The monkey patch allows us to log the data.

The key methods to patch are:

• open(): This method is called whenever a new AJAX call is initiated which I can then capture the method and url.
• setRequestHeader(): This method is called when the request header is set. I can patch it so that every time the method is called, I save the headers as hash.
• send(): This method is used to send data as part of the request. I can capture the request body from this patch.

Now, I can simply add addEventListener('load', event) to capture the response data.

There are a few resources on the web, such as this article, explaining how to monkey patch.

Execution Environment

With code ready for monkey patching, we have to execute it. Chrome Extension Tabs API has a method to execute the code: executeScript(). However, the code executed by executeScript() is executed in a different context than the code on the webpage, which is where the AJAX calls are made. Meaning the monkey patch won’t have any effect on the open website.

To learn more about how execution environment works, see this video on content scripts.

The Solution: Injected Script

Due to the isolated context, I had to take a different approach to successfully monkey patch the website. I had to use chrome.tabs.executeScript() to create a <script> tag injected on the website which would load the monkey patch code inside the website’s context.

There are several ways to add the code into the script tag. Since we also wanted to add some nice UI elements, we decided to put the monkey patch code and the UI related code into a script file, which the resourceUrl links to. If you do this, be sure to add the resource url to the web_accessible_resourcein the manifest. More information on Web Accessible Resources.

Known Issues We Punted On

CSS Leaking

Since the code is running as part of the web page, the UI we generated can be impacted by the CSS that is already on the page. We are using React and Material-UI library, which uses inline styling to minimize the CSS leaking. However, the CSS leaking still can happen on some websites.

Content Security Policy: content-src

Content security policies are another issue. If content-src is set to be very strict such as on GitHub, then it can blocks AJAX calls to servers that are not in a whitelist. Since we wanted to log and store the API calls to be retrieved from shareable workspace links on ApiRequest.io, we need to make POSTS to our own API. This allows the HTTP trace to be saved for 30-days as a reference during later debug.

The Chrome Extension execution environment is not fully subject to that restriction. There is a solution to modify the Content Security Policy which will be posted as a separate article.

At the moment, many websites are not using such stringent content security policies outside a few big ones like GitHub and Twitter. We are assuming that if you are using our Chrome Extension, then it probably is used on your own website. If so, then you can turn that restriction off temporarily.  if you have issues or feature requests for our Chrome extension.

Closing Thoughts

I had fun creating this Chrome Extension, so I hope you enjoy using it. As always, for any feature requests or issues when using the extension, just let me know.