DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • What Is API-First?
  • The Noticeable Shift in SIEM Data Sources
  • Mastering System Design: A Comprehensive Guide to System Scaling for Millions (Part 1)
  • Transforming Data Into JSON Structure With Spark: API-Based and SQL-Based Approaches

Trending

  • Mastering Fluent Bit: Installing and Configuring Fluent Bit on Kubernetes (Part 3)
  • Apache Doris vs Elasticsearch: An In-Depth Comparative Analysis
  • The Modern Data Stack Is Overrated — Here’s What Works
  • Infrastructure as Code (IaC) Beyond the Basics
  1. DZone
  2. Software Design and Architecture
  3. Integration
  4. Reverse Engineering CryptoPanic REST API

Reverse Engineering CryptoPanic REST API

Unleash the full potential of CryptoPanic's REST API with our guide and take your crypto game to the next level!

By 
Han Chiang user avatar
Han Chiang
·
Sep. 22, 23 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
3.8K Views

Join the DZone community and get the full member experience.

Join For Free

CryptoPanic is a news aggregator website for trending news based on social sentiment. It's a good website for keeping up to date on the latest news, as well as using it as a trading signal.

Its compact layout reminds me of hacker news. I was looking through the network requests and got intrigued after finding out that the main data is not in plaintext:


Posts request encrypted data


The server sends some sort of encoded data, and the client decodes it. After digging around the JS bundle, I managed to recover the plaintext data that is displayed on the website.

Note: This post is for educational purposes only.

The Reverse Engineering Process

1. Open the JS Bundle

The first thing is to find out which part of the JavaScript code is responsible for decoding this response. Begin by searching for and opening the JS bundle cryptopanic.min.xxxxx.js.

Chrome console js bundle


2. Locate the Part of the Code that Decodes the Data

Searching for "decrypt" immediately zooms in on this function called dc that does the core work of decoding the data.

dk() returns the encryption key, together with parameter t which is used as the Initialization Vector(IV), are passed through the AES algorithm with zero padding.

wordArrayToByteArray converts the decrypted response into a byte array and pako.default.inflate decompresses the data and converts them into JavaScript's utf-16.


Decrypt and inflate


After this step, the result is a JSON string of the API response. The JSON version looks like this:

Posts raw data


3. Transform the Raw Response Into a Usable Array of Objects

The last step is to transform the raw JSON into an array of objects so that it is easy to work with.

Posts normalized data


Revisiting the arguments passed to `dc` earlier, the first argument `t`(IV) is the first 16 characters of the module and some string or the CSRF token.

For posts API, t is news, and n is empty.

Decoding posts caller


For dashboard API, t is also news while n is rnlistrnlistrnlistrnlist.

Decoding dashboard caller


4. Final Output

This is the final output of the posts and dashboard response after decrypting, decompressing, and normalizing.

Posts normalized

Dashboad normalized

Cryptopanic home page highlighted


Try It Out Yourself

Here is a gist that you can use to try decoding the response yourself.

How To Use

  1. Create an .env file with CRYPTOPANIC_ENCRYPTION_KEY and CRYPTOPANIC_CSRF_TOKEN.
  2. Install the dependencies: npm install dotenv pako crypto-js
  3. Run node decrypt.js [post|dashboard]
JavaScript
 
require('dotenv').config();
const CryptoJS = require("crypto-js");
const pako = require('pako');
const fs = require('fs');
const path = require('path');

const ENCRYPTION_KEY = process.env.CRYPTOPANIC_ENCRYPTION_KEY;
const CSRF_TOKEN = process.env.CRYPTOPANIC_CSRF_TOKEN;

function wordToByteArray(t, e) {
    var n = [];
    return (
      e > 0 && n.push(t >>> 24),
      e > 1 && n.push((t >>> 16) & 255),
      e > 2 && n.push((t >>> 8) & 255),
      e > 3 && n.push(255 & t),
      n
    );
  }

function wordArrayToByteArray(t, e) {
    t.hasOwnProperty("sigBytes") &&
      t.hasOwnProperty("words") &&
      ((e = t.sigBytes), (t = t.words));
    for (var n, s = [], i = 0; e > 0; )
      (n = wordToByteArray(t[i], Math.min(4, e))),
        (e -= n.length),
        s.push(n),
        i++;
    return [].concat.apply([], s);
}

function normalizeDictList(t) {
    var e = [];
    return (
      t.l.forEach(function (n) {
        var s = {};
        t.k.forEach(function (t, e) {
          s[t] = n[e];
        }),
          e.push(s);
      }),
      e
    );
  }

// { words, sigBytes }
const key = CryptoJS.enc.Utf8.parse(ENCRYPTION_KEY);

// for posts
const postIvRaw = "news" + CSRF_TOKEN.substring(0, 12);
// for dashboard
const dashboardIvRaw = "newsrnlistrnlist";
const padding = CryptoJS.pad.ZeroPadding;

const postEncrypted = fs.readFileSync(path.join(__dirname, 'input', 'cryptopanic post data'), { encoding: 'utf-8' } );
const dashboardEncrypted = fs.readFileSync(path.join(__dirname, 'input', 'cryptopanic dashboard data'), { encoding: 'utf-8' });

const dataTypeMapping = {
    post: {
        iv: CryptoJS.enc.Utf8.parse(postIvRaw),
        encrypted: postEncrypted
    },
    dashboard: {
        iv: CryptoJS.enc.Utf8.parse(dashboardIvRaw),
        encrypted: dashboardEncrypted
    }
}

function decrypt(encrypted, key, iv, padding, dataType) {
    // { words, sigBytes }
    const decrypted = CryptoJS.AES.decrypt(encrypted, key, {
        iv,
        padding,
    });

    const byteArray = wordArrayToByteArray(decrypted);

    try {
        const inflated = pako.inflate(byteArray, { to: 'string' });
        const inflatedJson = JSON.parse(inflated);

        fs.writeFileSync(path.join(__dirname, 'decrypted', `${dataType}-inflatedRaw.json`), JSON.stringify(inflatedJson, undefined, 2));

        const normalized = normalizeDictList(inflatedJson);

        fs.writeFileSync(path.join(__dirname, 'decrypted', `${dataType}-normalized.json`), JSON.stringify(normalized, undefined, 2));

        return normalized;
    } catch(e) {
        console.log(e);
    }
}

// node decrypt.js
// dataType = 'post', 'dashboard'
if (require.main === module) {
    let [_, __, dataType] = process.argv;
    dataType = dataType ? dataType : 'post';

    const decrypted = decrypt(dataTypeMapping[dataType].encrypted, key, dataTypeMapping[dataType].iv, padding, dataType);
    console.log(decrypted);
}


API Engineering JSON REST Upload Data (computing)

Published at DZone with permission of Han Chiang. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • What Is API-First?
  • The Noticeable Shift in SIEM Data Sources
  • Mastering System Design: A Comprehensive Guide to System Scaling for Millions (Part 1)
  • Transforming Data Into JSON Structure With Spark: API-Based and SQL-Based Approaches

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!