A Gun is a Great equalizer: OpenJDK Hack vs. Class Encryption

The purpose of this article is to warn developers about futility of using class-file encryption obfuscators for application protection and uselessness of spending money on it.

All the protection methods are described in the fundamental work of Dmitry Leskov - Protect Your Java Code — Through Obfuscators And Beyond.

When we are using class file encryption method we assume that the byte-code is encrypted and when the application starts decrypted byte-code through custom ClassLoader or JVMTI-interface loads into JVM.

How to circumvent this kind of protection you can find in article of Dmitry Leskov, but today there are some products that have native component which interacts with JVM and monitors debug-mode and/or unauthorized agents. But despite all the assurances of the developers, these products does not protect your byte-code at all.

To show you vulnerability of all obfuscators which encrypts classes we will take some steps.

First of all, we should start encrypted application with option "-XX:+TraceClassLoading" and be sure that all encrypted class files are visible on this level of tracing.

Downloading the OpenJDK sources and patching it to dump encrypted classes to file.

For an experiment we are using Debian Linux 6.0.5 (Stable) and source bundle of OpenJDK7. If you are using different platform, please see OpenJDK Build README for instructions of how to build OpenJDK.

To minimize count of modifications if  "-XX:+TraceClassLoading" option is set JDK will dump all loaded classes in $WORKDIR/classes.dump file. Structure of dump file:

{
int lengthClassName,
byte[] className,
int lengthByteCode,
byte[] bytecode
}, 
{ next record … },
…

Let's prepare environment for build:

# apt-get install openjdk-6-jdk
# apt-get build-dep openjdk-6

Next, we have to download OpenJDK sources and our patch, which will add the following code into ClassFileParser::parseClassFile function, in hotspot/src/share/vm/classfile/classFileParser.cpp:

      // dumping class bytecode
      // dump file format:
      // length of the class name - 4 bytes
      // class name
      // length of the class bytecode - 4 bytes
      // byte code
      // ... next class ...
	  ClassFileStream* cfs = stream();
	  FILE * pFile;
	  int length = cfs->length();
	  int nameLength = strlen(this_klass->external_name());
	  pFile = fopen("classes.dump","ab");
	  // size of the class name
	  fputc((int)((nameLength >> 24) & 0XFF), pFile );
	  fputc((int)((nameLength >> 16) & 0XFF), pFile );
	  fputc((int)((nameLength >> 8) & 0XFF), pFile );
	  fputc((int)(nameLength & 0XFF), pFile );
      // class name
	  fwrite (this_klass->external_name() , 1, nameLength, pFile );
	  // size of the class bytecode
	  fputc((int)((length >> 24) & 0XFF), pFile );
	  fputc((int)((length >> 16) & 0XFF), pFile );
	  fputc((int)((length >> 8) & 0XFF), pFile );
	  fputc((int)(length & 0XFF), pFile );
      // class bytecode
	  fwrite (cfs->buffer() , 1 , length, pFile );
	  fclose(pFile);		

Let's try to build unmodified JDK to be sure that it builds correctly:

# export LANG=C ALT_BOOTDIR=/usr/lib/jvm/java-6-openjdk ALLOW_DOWNLOADS=true
# make sanity && make 

Apply patch and start build:

# cd $OPENJDK_SRC
# patch -p1 < $PATH_TO_PATCH_FILE
# make

Next, change dir to bin of builded JVM: $OPENJDK_SRC/build/linux-i586/j2re-image/bin/

To test work of patched JRE start java with -XX:+TraceClassLoading:

# ./java -XX:+TraceClassLoading

If all correct you will see classes.dump with all classes which JRE loads in start phase.

And now the most interesting moment. Download some encrypted application, for example trial version of obfuscator with class encryption. I will not mention for obvious reasons, specific names, it is enough to search on Google for the key «byte-code encryption». 

Let's say we have SomeClassGuard.jar. Inside SomeClassGuard.jar in the hierarchy com/ ****/ someclassguard/engine there are class-encrypted files, you can see it for yourself using any decompiler or look at the file header in the HEX-viewer.

Start the SomeClassGuard.jar:

# ./java -XX:+TraceClassLoading -jar SomeClassGuard.jar 

Ok, we have classes.dump now, but we need to parse this file. Let's write java program for that particular case:

package openjdkmod;

import java.io.DataInputStream;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;

/**
* Classes dump format extractor class.
* Author Ivan Kinash kinash@licel.ru
*/
public class ClassesDumpExractor {

   /**
    * Extract contents classes.dump to specified dir
    */
   public static void main(String[] args) throws
FileNotFoundException, IOException {
       if (args.length != 2) {
           System.err.println("Usage openjdkmod.ClassesDumpExtractor
 ");
           System.exit(-1);
       }
       File classesDumpFile = new File(args[0]);
       if (!classesDumpFile.exists()) {
           System.err.println("Source file: " + args[0] + " not found!");
           System.exit(-1);
       }
       File outDir = new File(args[1]);
       if (!outDir.exists()) {
           outDir.mkdirs();
       }
       DataInputStream din = new DataInputStream(new
FileInputStream(classesDumpFile));
       while (true) {
           try {
               int classNameLength = din.readInt();
               byte[] classNameBytes = new byte[classNameLength];
               din.readFully(classNameBytes);
               String className = new String(classNameBytes);
               System.out.println("className:" + className);
               int classLength = din.readInt();
               byte[] classBytes = new byte[classLength];
               din.readFully(classBytes);
               File parentDir = className.indexOf(".")>0?new
File(outDir, className.substring(0,className.lastIndexOf(".")).replace(".",
File.separator)):outDir;
               if(!parentDir.exists()) parentDir.mkdirs();
               File outFile = new File(parentDir,
(className.indexOf(".")>0?className.substring(className.lastIndexOf(".")+1):className)+".class");
               FileOutputStream outFos = new FileOutputStream(outFile);
               outFos.write(classBytes);
               outFos.close();
           } catch (EOFException e) {
               din.close();
               return;
           }
       }


   }
}

Let's execute compiled class like this:

# java openjdkmod.ClassesDumpExractor classes.dump dump_directory

On the output we'll get the directory with decrypted class files.

Conclusion

Class encryption is senseless, danger and expensive exercise.

If you want to protect your code there are few ways to do that:

• Use Java Byte-code to Native-code Compilers
• Combine classical obfuscator with string encryption obfuscators

For the super security use external devices which supports secure storage and internal byte-code execution.

You can use the method described above for debugging various applications, when you want to investigate which byte-code loads into JVM in runtime.

Note 1:  You can reach the same result without any modification of JDK, using sun.misc.Unsafe class if you aware about class storage methods inside JVM.

Note 2: The author is not responsible for the misuse of information from the article.

Note 3: Original source for the picture is: http://it.wikipedia.org/wiki/File:Netbeans-Duke.png