DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Enhanced API Security: Fine-Grained Access Control Using OPA and Kong Gateway
  • HTTP API: Key Skills for Smooth Integration and Operation (Part 1)
  • Rate Limiting Strategies for Efficient Traffic Management
  • Unleashing the Power of GPT: A Comprehensive Guide To Implementing OpenAI’s GPT in ReactJS

Trending

  • Mastering Fluent Bit: Installing and Configuring Fluent Bit on Kubernetes (Part 3)
  • Streamlining Event Data in Event-Driven Ansible
  • AI Meets Vector Databases: Redefining Data Retrieval in the Age of Intelligence
  • Recurrent Workflows With Cloud Native Dapr Jobs
  1. DZone
  2. Data Engineering
  3. Databases
  4. Capturing AJAX API Requests From Arbitrary Sites With a Chrome Extension

Capturing AJAX API Requests From Arbitrary Sites With a Chrome Extension

There were no suitable Chrome extensions to monitor HTTP requests from AJAX calls — so we decided to create one ourselves.

By 
Derric Gilling user avatar
Derric Gilling
DZone Core CORE ·
Mar. 08, 17 · Opinion
Likes (1)
Comment
Save
Tweet
Share
14.8K Views

Join the DZone community and get the full member experience.

Join For Free

I assumed there would be a lot of Chrome extensions to monitor HTTP requests from AJAX calls, but the few that we found (such as the Postman Intercept Chrome Extension) seem to only capture the request only and not the response. Even though Chrome DevTools has the network tab, it's hard to share those captured HTTP traces with teammates.

Our company, Moesif API Analytics, is very API-driven internally and externally. We divide responsibilities among API boundries, but we constantly have to copy and paste the HTTP headers and JSON payload from the DevTools into email or Slack when debugging API issues. Sending multi-megabyte *.har files through Slack is just as cumbersome. In addition to not being easy to share, inspecting HTTP traffic in Chrome DevTools is not ideal because there is no easy way to filter the API Calls or inspect JSON payloads. The network tab is designed for traditional HTML websites rather than modern single-page apps and APIs that return JSON.

Due to this time sink, we decided to build a Chrome Extension to make capturing and debugging these AJAX requests (and responses) from any website real easy. The extension is designed for REST APIs in mind, such as those powering single-page apps. However, some monkey business was needed to do the capturing that went around the Chrome Extension APIs due to limitations in Chrome’s APIs.

Issues With WebRequest Approach

Chrome WebRequest API is part of the Chrome Extension set of APIs.

Initially, I looked at the Chrome’s WebRequest APIs to capture API calls from an open browser tab. We leveraged those APIs before for our CORS and Origin Changer. It seemed to be the most natural place to capture the data.

However, we ran into a bug where the WebRequest API doesn’t expose an interface to read the response body. See the issue here on Chromium.

While we were surprised by the lack of a response body getters, we now understand a possible reason why most other extensions do not capture the responses. Our requirement was to capture the response so that they could be shared with team members. Without responses, debug would be painful and resolving issues would be slow.

XmlHttpRequest Event Listeners Approach

Since XmlHttpRequest is used by all browsers for AJAX calls, I leveraged this to monitor the API calls through monkey patching.

There are several XmlHttpRequest listeners for hooking into:

  • loadstart.
  • progress.
  • abort.
  • error.
  • load.
  • timeout.
  • loadend.
  • readystatechange.

From the listener callbacks, I can get the event target, which is the XmlHttpRequest object itself. Looking at the XmlHttpRequest api documentation, I realized it has exact the opposite problem as WebRequest API. The object has interfaces to get the HTTP response but not the original request. There is a setRequestHeaders(), but no getter method to pair with it to get the request HTTP headers. In fact, it doesn’t even have a public method or property to get the original URL or path.

XmlHttpRequest Monkey Patch

To truly monitor all the data (i.e. request headers/body and response headers/body), I’d have to monkey patch it. The monkey patch allows us to log the data.

The key methods to patch are:

  • open(): This method is called whenever a new AJAX call is initiated which I can then capture the method and url.
  • setRequestHeader(): This method is called when the request header is set. I can patch it so that every time the method is called, I save the headers as hash.
  • send(): This method is used to send data as part of the request. I can capture the request body from this patch.

Now, I can simply add addEventListener('load', event) to capture the response data.

There are a few resources on the web, such as this article, explaining how to monkey patch.

Execution Environment

With code ready for monkey patching, we have to execute it. Chrome Extension Tabs API has a method to execute the code: executeScript(). However, the code executed by executeScript() is executed in a different context than the code on the webpage, which is where the AJAX calls are made. Meaning the monkey patch won’t have any effect on the open website.

To learn more about how execution environment works, see this video on content scripts.

The Solution: Injected Script

Due to the isolated context, I had to take a different approach to successfully monkey patch the website. I had to use chrome.tabs.executeScript() to create a <script> tag injected on the website which would load the monkey patch code inside the website’s context.

There are several ways to add the code into the script tag. Since we also wanted to add some nice UI elements, we decided to put the monkey patch code and the UI related code into a script file, which the resourceUrl links to. If you do this, be sure to add the resource url to the web_accessible_resourcein the manifest. More information on Web Accessible Resources.

Known Issues We Punted On

CSS Leaking

Since the code is running as part of the web page, the UI we generated can be impacted by the CSS that is already on the page. We are using React and Material-UI library, which uses inline styling to minimize the CSS leaking. However, the CSS leaking still can happen on some websites.

Content Security Policy: content-src

Content security policies are another issue. If content-src is set to be very strict such as on GitHub, then it can blocks AJAX calls to servers that are not in a whitelist. Since we wanted to log and store the API calls to be retrieved from shareable workspace links on ApiRequest.io, we need to make POSTS to our own API. This allows the HTTP trace to be saved for 30-days as a reference during later debug.

The Chrome Extension execution environment is not fully subject to that restriction. There is a solution to modify the Content Security Policy which will be posted as a separate article.

At the moment, many websites are not using such stringent content security policies outside a few big ones like GitHub and Twitter. We are assuming that if you are using our Chrome Extension, then it probably is used on your own website. If so, then you can turn that restriction off temporarily.  if you have issues or feature requests for our Chrome extension.

Closing Thoughts

I had fun creating this Chrome Extension, so I hope you enjoy using it. As always, for any feature requests or issues when using the extension, just let me know.

API AJAX Requests

Published at DZone with permission of Derric Gilling. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Enhanced API Security: Fine-Grained Access Control Using OPA and Kong Gateway
  • HTTP API: Key Skills for Smooth Integration and Operation (Part 1)
  • Rate Limiting Strategies for Efficient Traffic Management
  • Unleashing the Power of GPT: A Comprehensive Guide To Implementing OpenAI’s GPT in ReactJS

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: