Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

The Magic Behind Burp, ZAP, and Other Proxies

DZone 's Guide to

The Magic Behind Burp, ZAP, and Other Proxies

Need help creating your own proxy software?

· Security Zone ·
Free Resource

If you build web applications and care about security, you have probably used the Burp and ZAP proxy security tools. These tools perform dynamic analysis of live web applications to identify security vulnerabilities. Burp and ZAP can discover issues with your applications as you navigate through them via a browser. Essentially, it was configured as the "man in the middle" and was able to intercept all traffic between your browser and web application. Have you ever wondered how it is possible to intercept encrypted traffic over https? This article explains how it is done and provides a basic framework for creating your own proxy software.

To get started with Burp and ZAP (from now on, I'll refer to these as simply the "proxy"), you have to decide what port you want the proxy to listen on and configure your browser to use that port as a proxy. In Firefox, to use port 9000, your configuration might look like the following:

Image title

Note the Use, this proxy server for all protocols box should be checked. Your browser is now ready to send and receive data through a proxy. Let's now start to put together some code to handle these browser requests.

First, we'll need to set up a ServerSocket to listen for requests:

ServerSocketFactory serverSocketFactory = ServerSocketFactory.getDefault();
proxyServerSocket = serverSocketFactory.createServerSocket(LOCAL_PORT, 10000, InetAddress.getByName(LOCAL_INTERFACE));
ServerSocketHandler handler = new ServerSocketHandler(proxyServerSocket, false);
handler.start();


Here, we use a ServerSocketHandler thread to handle the requests:

class ServerSocketHandler extends Thread {
    ServerSocket serverSocket;
    boolean secure;

    ServerSocketHandler(ServerSocket serverSocket, boolean secure) {
        this.serverSocket = serverSocket;
        this.secure = secure;
    }

    public void run() {
        try {
            while (true) {
                Socket socket = serverSocket.accept();
                RequestHandler handler = new RequestHandler(socket);
                handler.start();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}


This thread loops indefinitely. When it receives a request, it creates a RequestHandler to process it. The first thing it does is read the request:

while ( (line = br.readLine()) != null) {

    if (first) {
        first = false;
        String chunks[] = line.split(" ");
        method = chunks[0];
        urlText = chunks[1];
    }

    buffer.append(line + "\r\n");
    if (line.equals(""))
        break;
}


In the code that follows, there is an if statement to determine if the request is proxied over https and an else clause that follows. Let's start with the else clause, which shows how non-https traffic is handled:

else if (urlText != null) {
    System.out.println("ATTEMPTING URL: " + urlText);
    URL url = new URL(urlText);

    Request request = new Request(Message.parseMessage(new ByteArrayInputStream(buffer.toString().getBytes()), true));

    SocketFactory socketFactory = SocketFactory.getDefault();
    int port = url.getPort();
    Socket targetSocket = socketFactory.createSocket(url.getHost(), port < 0 ? 80 : port);
    OutputStream targetOs = targetSocket.getOutputStream();
    targetOs.write(request.message.httpHeader.bytes);
    targetOs.write(request.message.body.rawBytes);
    targetOs.flush();

    InputStream targetIs = targetSocket.getInputStream();
    Response response = new Response(Message.parseMessage(targetIs, false));
    final RequestResponse rr = new RequestResponse(request, response);
    Thread thread = new Thread() {
        public void run() {
            try { DataManager.INSTANCE.add(rr); } 
            catch (IOException e) {
                e.printStackTrace();
            }
        }
    };
    thread.start();

    socket.getOutputStream().write(response.message.httpHeader.bytes);
    socket.getOutputStream().write(response.message.body.rawBytes);
    socket.getOutputStream().flush();
    socket.close();
}


In this block of code, we first create a Request object by parsing the bytes from the buffer object. buffer contains the contents of the HTTP request. The Request object is used to store an HTTP request. Its contents can be used to forward the original request to the target host and can be used to store the original request for our own purposes (just like Burp and ZAP).

Next, we create a socket using the target host address and send the request. We then read the response from the target host and parse that into a Response object. Now that we have a Request and Response, we can create a RequestResponse object and store it indefinitely using the DataManager class. We create a new thread to perform the actual storage because we don't want to wait for this task to complete before sending the response back to the client. Finally, we send the response back to the client.

There should be no major surprises discussed thus far. Just as you might imagine, a proxy intercepts a request and records it. It forwards the request to the target host and records the response before sending it back to the client. Proxying really only gets interesting when we start to examine what happens in an HTTPS request. You might think that the exact same process happens, except that it uses HTTPS. If you weren't interested in capturing the unencrypted data, there would be some truth to this, but otherwise, it gets quite complicated. You must consider that a client wishing to exchange encrypted data with example.com expects to receive a Certificate stamped with "example.com" by a Certificate Authority. If you have used Burp or ZAP before, then you probably already know that they generate their own certificates using their own Certificate Authority certificate, which must also be installed as a trusted certificate in your browser. If that wasn't clear, here it is in bullet form:

1. In PKI encryption, senders and receivers of encrypted data use certificates to identify themselves. Each certificate must be issued by a Certificate Authority. Certificate Authorities have certificates pre-installed in your browser and are automatically trusted.

2. Our proxy can't use these certificates because they are password protected. Hence, we create our own Certificate Authority certificate, which we can use to issue our own certificates.

3. Our homegrown Certificate Authority certificate must be installed in the browser in order to trust our own web site certificate.

Our first problem to tackle is generating a Certificate Authority certificate. I put together a utility class for this called CreateCertificateAuthorityUtil. It must be run once independent from the Proxy to create the certificate. It has a main method, which invokes the createAndExportSelfSignedCertificateTo (...)  method. Here they are:

public static void main(String[] args) throws Exception {
  createAndExportSelfSignedCertificateTo(CertUtil.KEY_PAIR_GENERATOR.generateKeyPair(), CA_KEYSTORE_FILE, CA_CERT_FILE);
}

public static X509Certificate createAndExportSelfSignedCertificateTo(KeyPair caKeyPair, String keyStoreLocation, String derFileLocation) throws Exception {
  X509Certificate caCert = CertUtil.createCertificate(CA_X500_NAME, CA_X500_NAME, caKeyPair.getPublic(), caKeyPair.getPrivate(), true);
  KeyStore ks = KeyStore.getInstance("JKS");
  ks.load(null, CA_KEYSTORE_PASSWORD.toCharArray());
  ks.setKeyEntry(CA_KEY_ALIAS, caKeyPair.getPrivate(), CA_KEYSTORE_PASSWORD.toCharArray(), new X509Certificate[]{caCert});

  try (PrintStream ps = new PrintStream(derFileLocation); 
    FileOutputStream fos = new FileOutputStream(keyStoreLocation);) {
    CertUtil.printCertificateTo(caCert, ps);
    ks.store(fos, CA_KEYSTORE_PASSWORD.toCharArray());
  }

  return caCert;
}


These methods reference some class-scoped variables, some of which are initialized in a static initializer block:

static final File TMP_DIRECTORY = new File(System.getProperty("java.io.tmpdir"));
static final File DEFAULT_CA_KEY_STORE_FILE = new File(TMP_DIRECTORY, "cakeystore.ks");
static final File DEFAULT_CA_CERT_FILE = new File(TMP_DIRECTORY, "cacert.der");

static final String CA_KEY_STORE_FILE_PROPERTY_NAME = "ca-keystore-file";
static final String CA_CERT_FILE_PROPERTY_NAME = "ca-cert-file";

public static final String CA_KEYSTORE_PASSWORD = "password";
public static final String CA_KEY_ALIAS = "ca_alias";

public static final String X500_CN_COMMON_NAME_VALUE = "Fake Certificate Authority Inc.";
public static final String X500_OU_ORGANIZATIONAL_UNIT_VALUE = "Fake Certificate Authority OrgUnit";
public static final String X500_O_ORGANIZATIONAL_VALUE = "Fake Certificate Authority Org";
public static final String X500_L_LOCALITY_VALUE = "Fake Certificate Authority City";
public static final String X500_ST_STATE_PROVINCE_NAME_VALUE = "Fake Certificate Authority State";
public static final String X500_C_COUNTRY_NAME_VALUE = "Fake Certificate Authority Country";

public static final String CA_KEYSTORE_FILE;
public static final String CA_CERT_FILE;

public static final X500Name CA_X500_NAME;

static {
  String value = System.getProperty(CA_KEY_STORE_FILE_PROPERTY_NAME);
  CA_KEYSTORE_FILE = value == null ? DEFAULT_CA_KEY_STORE_FILE.getAbsolutePath() : value;
  value = System.getProperty(CA_CERT_FILE_PROPERTY_NAME);
  CA_CERT_FILE = value == null ? DEFAULT_CA_CERT_FILE.getAbsolutePath() : value;
  try { CA_X500_NAME = CertUtil.createName(X500_CN_COMMON_NAME_VALUE, X500_OU_ORGANIZATIONAL_UNIT_VALUE, X500_O_ORGANIZATIONAL_VALUE, X500_L_LOCALITY_VALUE, X500_ST_STATE_PROVINCE_NAME_VALUE, X500_C_COUNTRY_NAME_VALUE); } 
  catch (IOException e) {
    throw new Error("Couldn't create Certificate Authority X500 Name", e);
  }
  System.out.println("CA Keystore file: " + CA_KEYSTORE_FILE + ", and CA Cert file: " + CA_CERT_FILE);
  System.out.println("CA X500 Name: " + CA_X500_NAME);
}


The static initializer is used set up the CA_KEYSTORE_FILE, CA_CERT_FILE, and CA_X500_Name constants. By making them static, they can be referenced by the proxy. The proxy will use the CA_KEYSTORE_FILE to get the Certificate Authority certificate's private key, which will be used to create the certificate to send to the browser to impersonate the real host. It also uses CA_X500_NAME to create the certificate.

Let's now take a closer look at the createAndExportSelfSignedCertificateTo(...)  method. The bulk of the code creates a KeyStore and stores the Certificate Authority certificate. However, the first line creates the Certificate Authority certificate by calling a method in the CertUtil utility class,  createCertificate(...). This method can be used to create Certificate Authority certificates or SSL certificates. It has a boolean isCertificateAuthority parameter to specify which type. 

Here is an overview of what it does:

  1. Creates an X509CertInfo object and gives it a serial and version number
  2. Defines the period for which it is valid
  3. Sets the subject and issuer
  4. Sets appropriate properties if creating a Certificate Authority
  5. Sets the public key and algorithm
  6. Signs it

If we now jump back to the createAndExportSelfSignedCertificateTo(...)  method, you'll note it not only stores the certificate in the KeyStore but also writes out the certificate to disk. This is so it can easily be imported by a browser. By default, the certificate is written to <java.io.tmpdir>/cacert.der. Let's now import it into FireFox.

First, click the Open Menu button and select Options:

Image title

Next, search for "Certificate" and click View Certificates:

Image title

In the resulting dialog, click the Authorities tab and then click the Import... button:

Image title

A file browser dialog will appear. Now, select the <java.io.tmpdir>/cacert.der file you created earlier. The following dialog will appear:

Image title

Click the first checkbox and click the OK button. You have now imported the Certificate Authority certificate into your browser.

We can now get back to understanding the proxy code, but first, let's consider how a proxy would be able to process an HTTPS request if it is encrypted. Specifically, if HTTPS functioned the same as HTTP, the proxy wouldn't know the target address because it is embedded in an encrypted HTTP request. To solve this problem, HTTPS uses a variation on HTTP. Instead of embedding the target host and port in the request headers, it sends a Connect request, which contains the URL and the HTTP method (i.e. GET, POST etc). The proxy will send back a receipt of the Connect request with a 200 response code. Here is the relevant code:

The next step is to create an SSLSocketFactory object to communicate with the real host over SSL:

SSLSocketFactory secureSocketFactory = CertUtil.getTunnelSSLSocketFactory(url.getHost());


You will notice the getTunnelSSLSocketFactory(...) method takes a parameter of the target host. We need this information to set up a fake certificate. Here is the code:

public static SSLSocketFactory getTunnelSSLSocketFactory(String hostname) throws UnrecoverableKeyException, KeyStoreException, NoSuchAlgorithmException, Exception {
  SSLSocketFactory factory = factoryMap.get(hostname);
  if (factory != null)
    return factory;

  try {
      SSLContext ctx = SSLContext.getInstance("TLS");
      KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
      KeyStore ks = generateHostKeystore(hostname);

      kmf.init(ks, PASSWORD.toCharArray());
      java.security.SecureRandom random = new java.security.SecureRandom();
      ctx.init(kmf.getKeyManagers(), null, random);
      SSLSocketFactory tunnelSSLFactory = ctx.getSocketFactory();

      factoryMap.put(hostname, tunnelSSLFactory);

      return tunnelSSLFactory;
  } 
  catch (Exception e) {
     throw new RuntimeException(e);
  }
}


The first line checks to see if we have set up an SSLSocketFactory for this host already. If so, we re-use it. The rest of this code is standard code for setting up the factory except for this line:

KeyStore ks = generateHostKeystore(hostname);


generateHostKeyStore(...)  is another utility method in CertUtil, which we require to set up the SSLSocketFactory. Here is the code:

public synchronized static KeyStore generateHostKeystore(String host) throws Exception {
  try (FileInputStream fis = new FileInputStream(CreateCertificateAuthorityUtil.CA_KEYSTORE_FILE); ) {
    KeyStore caKeyStore = KeyStore.getInstance("JKS");
    caKeyStore.load(fis, CreateCertificateAuthorityUtil.CA_KEYSTORE_PASSWORD.toCharArray());

    PrivateKey caPrivateKey = (PrivateKey) caKeyStore.getKey(CreateCertificateAuthorityUtil.CA_KEY_ALIAS, CreateCertificateAuthorityUtil.CA_KEYSTORE_PASSWORD.toCharArray());
    KeyPair hostKeyPair = KEY_PAIR_GENERATOR.generateKeyPair();
    X500Name hostName = createName(host, X500_HOST_OU_ORGANIZATIONAL_UNIT_VALUE, X500_HOST_O_ORGANIZATIONAL_VALUE, X500_HOST_L_LOCALITY_VALUE, X500_HOST_ST_STATE_PROVINCE_NAME_VALUE, X500_HOST_C_COUNTRY_NAME_VALUE);
    java.security.cert.Certificate cert = createCertificate(hostName, CreateCertificateAuthorityUtil.CA_X500_NAME, hostKeyPair.getPublic(), caPrivateKey, false);
    PrivateKey privateKey = hostKeyPair.getPrivate();

    KeyStore hostKeyStore = KeyStore.getInstance("JKS");
    hostKeyStore.load(null, PASSWORD.toCharArray());
    hostKeyStore.setKeyEntry(HOST_ALIAS, privateKey, PASSWORD.toCharArray(), new java.security.cert.Certificate[] { cert });

    return hostKeyStore;
  }
}


Here is an overview of the code:

  1. Extract the Certificate Authority private key from the Certificate Authority KeyStore
  2. Generate a new KeyPair for the new SSL certificate
  3. Create the X500Name for the host
  4. Create the host certificate using the CertUtil.createCertificate(...) method
  5. Store the host certificate in an in-memory KeyStore

Let's now go back to the proxy code. After we create the SSLSocketFactory, we have the following line:

SSLSocket sslSocket = (SSLSocket) secureSocketFactory.createSocket(socket, socket.getInetAddress().getHostAddress(), socket.getPort(), true);


This is a special version of the SecureSocketFactory.createSocket(...) method, which takes a socket a parameter. Essentially, it abstract the encryption/decryption for us, so we can read and write unencrypted data to the socket that the browser is connected to.

The next few lines tell the socket to behave in client mode and initiate the SSL handshake:

sslSocket.setUseClientMode(false);
try {
  sslSocket.startHandshake();
} catch (Exception e) {
System.err.println("Error on URL: " + urlText);
  throw new RuntimeException(e);
}


The rest of the code in this method is essentially the same as the code for handling HTTP in the else clause. You can now run the Proxy class and start proxying!

You should note that this code is only proof-of-concept quality and would need a vast amount of improvement before being production ready. For example, it only handles 200 response codes. It will throw exceptions for other response codes.

You may also want to note the following:

  1. The code uses restricted access APIs from the JRE. In order to reference them in Eclipse, you will have to modify how Eclipse treats these APIs. Specifically, go to Java Compiler Errors/Warnings panel and change the Forbidden Reference (access rules) setting from Error to Warning or less.
  2. When testing your proxy code, it can be a little bit overwhelming when visiting a real web page. Most web pages don't generate a single request. With images, style sheets, and scripts, a single web page can generate many requests. It can be confusing the debug your proxy code when multiple requests are being created concurrently. To skirt this issue, I created a ProxyTest class, which uses HttpURLConnection to send a single request. Here is the code:
public static void main(String[] args) throws IOException {
  java.net.Proxy proxy = 
    new java.net.Proxy(java.net.Proxy.Type.HTTP, new InetSocketAddress(Proxy.DEFAULT_LOCAL_INTERFACE, Proxy.DEFAULT_LOCAL_PORT));

//    URL url = new URL("https://www.google.ca/?q=asdf");
//    URL url = new URL("https://www.google.ca/");
//    URL url = new URL("https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png");
    URL url = new URL("http://www.tutorialspoint.com/");

    HttpURLConnection.setFollowRedirects(false);
    HttpURLConnection c = (HttpURLConnection) url.openConnection(proxy);

    InputStream is = c.getInputStream();
    IOUtil.copy(is, System.out);
}


You should be aware that HttpURLConnection has setFollowRedirects(...) set to false. It will, otherwise, generate multiple requests for a redirect response code, which can be confusing.

You can find the full source code for this project on GitHub and the javadocs here.

Topics:
java security ,security ,serversocket ,proxies ,zap ,burp ,https ,ketstore ,certficate security

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}