Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Supporting Encrypted Content-Encoding in HttpClient (Part 1 of 2)

DZone's Guide to

Supporting Encrypted Content-Encoding in HttpClient (Part 1 of 2)

What if you need to encrypt your content in a way that the server side can only store it and whoever will request it in the future will need a proper key to decrypt it?

· Security Zone
Free Resource

Discover an in-depth knowledge about the different kinds of iOS hacking tools and techniques with the free iOS Hacking Guide from Security Innovation.

When you hear HTTP and encryption typically your first thought is SSL. SSL allows for encoding entire communications between client and server, but what if there is a need to encrypt the content in a way that the other side can only store it and whoever will request it in the future will need a proper key to decrypt it? The "Encrypted Content-Encoding for HTTP" (in version 07 at the moment of writing this) aims at providing a standard solution for such scenarios. In this post and the following post, I'm going to show how it can be used with HttpClient.

The "aes128gcm" Encoding

The "Encrypted Content-Encoding for HTTP" introduces a new value for Content-Encoding header - aes128gcm. This encoding allows for transferring encrypted data together with information necessary to decrypt them when somebody knows the key. The encoded body consists of a coding header and encrypted content represented by a number of fixed size encrypted records ( the last record can be smaller than others and there is a basic mechanism for preventing removal or reordering of records). For encryption purposes, the AES in Galois/Counter mode with a 128-bit key is being used.

Adding Encoding Capability to HttpClient

What I want to achieve is support for aes128gcm encoding on top of any content type. From the HttpClient perspective, it seems like something that can be easily achieved with a custom HttpContent which would serve as a wrapper over other ones.

public sealed class Aes128GcmEncodedContent : HttpContent
{
    private readonly HttpContent _contentToBeEncrypted;
    private readonly byte[] _key;
    private readonly string _keyId;
    private readonly int _recordSize;

    public Aes128GcmEncodedContent(HttpContent contentToBeEncrypted, byte[] key, string keyId, int recordSize)
    {
        _contentToBeEncrypted = contentToBeEncrypted;

        Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
        Headers.ContentEncoding.Add("aes128gcm");
    }
}


The key, key identifier, and record size are parameters which control the encoding algorithm so they will be needed. I'm also setting Content-Encoding header to aes128gcm and Content-Type to application/octet-stream. The second one is suggested by the specification but there is also an option for skipping the Content-Type header - the goal is to prevent exposure of the original Content-Type. In order to add the encoding I've decided to override the SerializeToStreamAsync method which should allow me to perform work over streams whenever possible.

public sealed class Aes128GcmEncodedContent : HttpContent
{
    ...

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        Stream streamToBeEncrypted = await _contentToBeEncrypted.ReadAsStreamAsync();

        await Aes128GcmEncoding.EncodeAsync(streamToBeEncrypted, stream, _key, _keyId, _recordSize);
    }

    protected override bool TryComputeLength(out long length)
    {
        length = 0;

        return false;
    }
}

Implementing the Encoding Algorithm

At a high level, the encoding algorithm consists of just a few steps, so it is simpler to show the implementation and then describe each of the steps.

public static class Aes128GcmEncoding
{
    public static async Task EncodeAsync(Stream source, Stream destination, byte[] key, string keyId, int recordSize)
    {
        ...

        if ((key == null) || (key.Length != 16))
            throw new ArgumentException(
                $"The '{nameof(key)}' parameter must be 16 octets long.", nameof(key));

        if (recordSize < 18)
            throw new ArgumentException(
                $"The '{nameof(recordSize)}' parameter must be at least 18.", nameof(recordSize));

        byte[] salt = GenerateSalt();

        byte[] pseudorandomKey = HmacSha256(salt, key);
        byte[] contentEncryptionKey = GetContentEncryptionKey(pseudorandomKey);

        await WriteCodingHeaderAsync(destination, salt, keyId, recordSize);

        await EncryptContentAsync(source, destination, recordSize, pseudorandomKey, contentEncryptionKey);
    }
}


First, some validations must be performed (I've skipped the null checks for brevity):

  • The key must be provided and must have exactly 128 bits. This comes from the encryption algorithm being used.
  • The record size must be at least 18 bytes. This comes from the fact that the encryption algorithm produces output longer by 16 bytes from the source and a delimiter byte is required (so if the record size will be 18 bytes it means we can put exactly 1 byte of data into it).

If the provided parameters are valid the salt can be generated.

public static class Aes128GcmEncoding
{
    ...

    private static readonly SecureRandom _secureRandom = new SecureRandom();

    ...

    private static byte[] GenerateSalt()
    {
        byte[] salt = new byte[16];
        _secureRandom.NextBytes(salt, 0, 16);

        return salt;
    }

    ...
}


The salt will be used to generate the content encryption key which will serve as the actual key for encryption. This is needed in order to prevent key exposure in cases when different content is being encrypted using the same keying material - without the random part, this wouldn't be safe. In order to derive the content encryption key from salt and key, the HKDF algorithm must be used. The first step is calculating a pseudorandom key which is the result of the HMAC SHA-256 hash of salt with the key.

public static class Aes128GcmEncoding
{
    ...

    private static byte[] HmacSha256(byte[] key, byte[] value)
    {
        byte[] hash = null;

        using (HMACSHA256 hasher = new HMACSHA256(key))
        {
            hash = hasher.ComputeHash(value);
        }

        return hash;
    }

    ...
}


With a pseudorandom key, the content encryption key can be calculated as truncated to 16 bytes HMAC SHA-256 hash of the pseudorandom key with the content encryption key info parameter. The content encryption key info parameter is an ASCII-encoded Content-Encoding: aes128gcm string terminated by 0x00 and 0x01 bytes.

public static class Aes128GcmEncoding
{
    ...

    private static readonly byte[] _contentEncryptionKeyInfoParameter;

    ...

    static Aes128GcmEncoding()
    {
        _contentEncryptionKeyInfoParameter = GetInfoParameter("Content-Encoding: aes128gcm");
    }

    ...

    private static byte[] GetInfoParameter(string infoParameterString)
    {
        byte[] infoParameter = new byte[infoParameterString.Length + 2];
        Encoding.ASCII.GetBytes(infoParameterString, 0, infoParameterString.Length, infoParameter, 0);

        infoParameter[infoParameter.Length - 1] = 1;

        return infoParameter;
    }

    private static byte[] GetContentEncryptionKey(byte[] pseudorandomKey)
    {
        byte[] contentEncryptionKey = HmacSha256(pseudorandomKey, _contentEncryptionKeyInfoParameter);
        Array.Resize(ref contentEncryptionKey, 16);

        return contentEncryptionKey;
    }

    ...
}


Those are all the prerequisites and now the body of the message can be created. First, the header must be written and then the encrypted records.

Writing the Coding Header

The coding header has four fields: salt, record size, key identifier size, and key identifier. The size of the header is variable as it depends on the length of the key identifier; this is why the key identifier size must be provided. The size is being kept as a single byte which means that key identifier can take up to 255 bytes. The key identifier is expected to be a UTF-8 encoded string, so with a simple method we can transform it to an array of bytes and verify the length.

public static class Aes128GcmEncoding
{
    ...

    private static byte[] GetKeyIdBytes(string keyId)
    {
        byte[] keyIdBytes = String.IsNullOrEmpty(keyId) ? new byte[0] : Encoding.UTF8.GetBytes(keyId);
        if (keyIdBytes.Length > Byte.MaxValue)
        {
            throw new ArgumentException($"The '{nameof(keyId)}' parameter is too long.", nameof(keyId));
        }

        return keyIdBytes;
    }

    ...
}


The second thing which needs to be transformed into a byte array is the record size. There are 4 bytes reserved for record size in the header which means that it can store an unsigned 32-bit integer, but my implementation is limited to the signed integer which makes it simpler (especially as some of the methods are available only over arrays which can't be bigger than 2GB).

public static class Aes128GcmEncoding
{
    ...

    private static byte[] GetRecordSizeBytes(int recordSize)
    {
        byte[] recordSizeBytes = BitConverter.GetBytes(recordSize);
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(recordSizeBytes);
        }

        return recordSizeBytes;
    }

    ...
}


The header needs to be put together and written to the target stream.

public static class Aes128GcmEncoding
{
    ...

    private static async Task WriteCodingHeaderAsync(Stream destination, byte[] salt, string keyId, int recordSize)
    {
        byte[] keyIdBytes = GetKeyIdBytes(keyId);
        byte[] recordSizeBytes = GetRecordSizeBytes(recordSize);

        byte[] codingHeader = new byte[21 + keyIdBytes.Length];

        salt.CopyTo(codingHeader, 0);
        recordSizeBytes.CopyTo(codingHeader, 16);
        codingHeader[20] = (byte)keyIdBytes.Length;
        keyIdBytes.CopyTo(codingHeader, 21);

        await destination.WriteAsync(codingHeader, 0, codingHeader.Length);
    }

    ...
}

Encrypting the Content and Writing the Records

The implementation of AES GCM is beyond the scope of a blog post and I'm not even going to attempt doing that, instead, I've chosen to use Bouncy Castle. As mentioned previously, the encrypted content must be represented by fixed size records (last can be shorter), which means that source data needs to be properly split. Also, the last record must be properly detected because it should be terminated with 0x02 byte while all the previous ones should be terminated with 0x01 byte. Detecting the last record is easy if the content doesn't split equally between records as the final read will return a smaller number of bytes than requested. The situation when content does split equally is a little bit more tricky - it requires checking if the stream can be further read before writing the record to the target. The safest way to do that seems to be reading a single byte in advance and then adding it to the next record. I've started the implementation with a helper method which reads the required number of bytes, prepends it with that "peeked" byte, and initially sets the delimiter based on the number of returned bytes.

public static class Aes128GcmEncoding
{
    ...

    private static async Task GetPlainTextAsync(Stream source, int recordDataSize, byte? peekedByte)
    {
        int readDataSize;
        byte[] plainText = new byte[recordDataSize + 1];

        if (peekedByte.HasValue)
        {
            plainText[0] = peekedByte.Value;
            readDataSize = (await source.ReadAsync(plainText, 1, recordDataSize - 1)) + 1;
        }
        else
        {
            readDataSize = await source.ReadAsync(plainText, 0, recordDataSize);
        }

        if (readDataSize == recordDataSize)
        {
            plainText[plainText.Length - 1] = 1;
        }
        else
        {
            Array.Resize(ref plainText, readDataSize + 1);
            plainText[plainText.Length - 1] = 2;
        }

        return plainText;
    }

    ...
}


The number of bytes to read is being calculated from the record size by subtracting the previously mentioned overhead (17 bytes). This allows for putting the core of the encryption routine together.

public static class Aes128GcmEncoding
{
    ...

    private static async Task EncryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;
        int recordDataSize = recordSize - 17;

        byte[] plainText = null;
        int? peekedByte = null;

        do
        {
            plainText = await GetPlainTextAsync(source, recordDataSize, (byte?)peekedByte);

            if (plainText[plainText.Length - 1] != 2)
            {
                peekedByte = source.ReadByte();
                if (peekedByte == -1)
                {
                    plainText[plainText.Length - 1] = 2;
                }
            }

            // TODO: Encrypt and write the record
        }
        while (plainText[plainText.Length - 1] != 2);
    }

    ...
}


The AES GCM requires one more parameter which needs to be calculated - nonce. The aes128gcm encoding uses nonce additionally for removal and reordering protection by performing a XOR with a record sequence number as the last step. The first argument for that XOR is the result of the same HKDF function as the one discussed in the context of the content encryption key, the difference, here, is the key info parameter (Content-Encoding: nonce) and length (12 bytes).

public static class Aes128GcmEncoding
{
    ...

    private static readonly byte[] _nonceInfoParameter;

    ...

    static Aes128GcmEncoding()
    {
        ...
        _nonceInfoParameter = GetInfoParameter("Content-Encoding: nonce");
    }

    ...

    private static byte[] GetNonce(byte[] pseudorandomKey, ulong recordSequenceNumber)
    {
        byte[] nonce = HmacSha256(pseudorandomKey, _nonceInfoParameter);
        Array.Resize(ref nonce, 12);

        byte[] recordSequenceNumberBytes = BitConverter.GetBytes(recordSequenceNumber);
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(recordSequenceNumberBytes);
        }
        int leadingNullBytesCount = 12 - recordSequenceNumberBytes.Length;

        for (int i = 0; i < leadingNullBytesCount; i++)
        {
            nonce[i] = (byte)(nonce[i] ^ 0);
        }

        for (int i = 0; i < recordSequenceNumberBytes.Length; i++)
        {
            nonce[leadingNullBytesCount + i] =
                (byte)(nonce[leadingNullBytesCount + i] ^ recordSequenceNumberBytes[i]);
        }

        return nonce;
    }

    ...
}


With nonce calculated every record can be encrypted and written to the target.

public static class Aes128GcmEncoding
{
    ...

    private static async Task EncryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;
        int recordDataSize = recordSize - 17;

        byte[] plainText = null;
        int? peekedByte = null;

        do
        {
            plainText = await GetPlainTextAsync(source, recordDataSize, (byte?)peekedByte);

            if (plainText[plainText.Length - 1] != 2)
            {
                peekedByte = source.ReadByte();
                if (peekedByte == -1)
                {
                    plainText[plainText.Length - 1] = LAST_RECORD_DELIMITER;
                }
            }

            aes128GcmCipher.Reset();
            AeadParameters aes128GcmParameters = new AeadParameters(new KeyParameter(contentEncryptionKey),
                128, GetNonce(pseudorandomKey, recordSequenceNumber));
            aes128GcmCipher.Init(true, aes128GcmParameters);

            byte[] cipherText = new byte[aes128GcmCipher.GetOutputSize(plainText.Length)];
            int lenght = aes128GcmCipher.ProcessBytes(plainText, 0, plainText.Length, cipherText, 0);
            aes128GcmCipher.DoFinal(cipherText, lenght);

            await destination.WriteAsync(cipherText, 0, cipherText.Length);
        }
        while (plainText[plainText.Length - 1] != 2);
    }

    ...
}


Take it for a Spin

With all the pieces in place, the Aes128GcmEncodedContent can be used to make an actual request.

using (HttpClient encryptedContentEncodingClient = new HttpClient())
{
    HttpContent contentToBeEncrypted = new StringContent("I am the walrus", Encoding.UTF8);

    byte[] key = Convert.FromBase64String("yqdlZ+tYemfogSmv7Ws5PQ==");
    HttpContent encryptedContent = new Aes128GcmEncodedContent(contentToBeEncrypted, key, null, 4096);

    await encryptedContentEncodingClient.PostAsync("<URL>", encryptedContent);
}


Both Aes128GcmEncodedContent and Aes128GcmEncoding can be grabbed directly from here.

In my next post, I'm going to focus on the decoding part of all this. 

Learn about the importance of a strong culture of cybersecurity, and examine key activities for building – or improving – that culture within your organization.

Topics:
security ,encryption ,httpclient

Published at DZone with permission of Tomasz Pęczek. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}