Supporting Encrypted Content-Encoding in HttpClient (Part 2 of 2)

In my previous post, I've shown how HttpClient can be extended with payload encryption capabilities by providing support for aes128gcm encoding. In this post, I'm going to extend theAes128GcmEncoding class with decoding capabilities.

Decoding at the High Level

It shouldn't be a surprise that decoding is mostly about doing the opposite of encoding. This is why the DecodeAsync method is very similar to EncodeAsync.

public static class Aes128GcmEncoding
{
    public static async Task DecodeAsync(Stream source, Stream destination, Func<string, byte[]> keyLocator)
    {
        // Validation skipped for brevity
        ...

        CodingHeader codingHeader = await ReadCodingHeaderAsync(source);

        byte[] pseudorandomKey = HmacSha256(codingHeader.Salt, keyLocator(codingHeader.KeyId));
        byte[] contentEncryptionKey = GetContentEncryptionKey(pseudorandomKey);

        await DecryptContentAsync(source, destination,
            codingHeader.RecordSize, pseudorandomKey, contentEncryptionKey);
    }
}

The keyLocator parameter is a simple way for delegating the key management responsibility to the caller, the implementation expects a method for retrieving it based on key identifier without going into any further details. I have also decided to introduce a class for the coding header properties in order to make the code more readable.

Retrieving the Coding Header

As we already know the coding header contains three fields with constant length (salt, record size and, key identifier length) and one with variable length (zero in extreme case). They can be retrieved one by one. The important thing is to validate the presence and size of every field, for this purpose I've split the reading into several smaller methods. Also, the record size must be additionally validated as this implementation support is a smaller value than what is allowed by the specification.

public static class Aes128GcmEncoding
{
    private static async Task<byte[]> ReadCodingHeaderBytesAsync(Stream source, int count)
    {
        byte[] bytes = new byte[count];
        int bytesRead = await source.ReadAsync(bytes, 0, count);
        if (bytesRead != count)
            throw new FormatException("Invalid coding header.");

        return bytes;
    }

    private static async Task<int> ReadRecordSizeAsync(Stream source)
    {
        byte[] recordSizeBytes = await ReadCodingHeaderBytesAsync(source, RECORD_SIZE_LENGTH);

        if (BitConverter.IsLittleEndian)
            Array.Reverse(recordSizeBytes);
        uint recordSize = BitConverter.ToUInt32(recordSizeBytes, 0);

        if (recordSize > Int32.MaxValue)]
            throw new NotSupportedException($"Maximum supported record size is {Int32.MaxValue}.");

        return (int)recordSize;
    }

    private static async Task<string> ReadKeyId(Stream source)
    {
        string keyId = null;

        int keyIdLength = source.ReadByte();

        if (keyIdLength == -1)
            throw new FormatException("Invalid coding header.");

        if (keyIdLength > 0)
        {
            byte[] keyIdBytes = await ReadCodingHeaderBytesAsync(source, keyIdLength);
            keyId = Encoding.UTF8.GetString(keyIdBytes);
        }

        return keyId;
    }

    private static async Task<CodingHeader> ReadCodingHeaderAsync(Stream source)
    {
        return new CodingHeader
        {
            Salt = await ReadCodingHeaderBytesAsync(source, SALT_LENGTH),
            RecordSize = await ReadRecordSizeAsync(source),
            KeyId = await ReadKeyId(source)
        };
    }
}

With the coding header retrieved the content can be decrypted.

Decrypting the Content and Retrieving the Payload

The pseudorandom key and content encryption key should be calculated in exactly the same way as during encryption. With those, the records can be read and decrypted. The operation should be done record by record (as mentioned in the previous post the nonce guards the order) until the last record is reached. Here, reaching the last record means not only reaching the end of content but must be confirmed by that last record being delimited with a 0x02 byte.

The tricky part is extracting the data from the record. In order to do that we need to detect the location of the delimiter and make sure it meets all the requirements. All the records must be of an equal length (except the last one) but they don't have to contain the same amount of data, as there can be padding consisting of any number of 0x00 bytes at the end. This is something which I haven't included into the encryption implementation but must be correctly handled here. So the delimiter should be the first byte from the end which has a value that is not 0x00. As explained in the previous post there are two valid delimiters: 0x01 (for all the records except the last one) and 0x02 (for the last record). Any other delimiter means that record is invalid, also a record which contains only padding is invalid. The below method ensures all those conditions are met:

public static class Aes128GcmEncoding
{
    private static int GetRecordDelimiterIndex(byte[] plainText, int recordDataSize)
    {
        int recordDelimiterIndex = -1;
        for (int plaintTextIndex = plainText.Length - 1; plaintTextIndex >= 0; plaintTextIndex--)
        {
            if (plainText[plaintTextIndex] == 0)
                continue;

            if ((plainText[plaintTextIndex] == RECORD_DELIMITER)
                || (plainText[plaintTextIndex] == LAST_RECORD_DELIMITER))
            {
                recordDelimiterIndex = plaintTextIndex;
            }

            break;
        }

        if ((recordDelimiterIndex == -1)
            || ((plainText[recordDelimiterIndex] == RECORD_DELIMITER)
                && ((plainText.Length -1) != recordDataSize)))
        {
            throw new FormatException("Invalid record delimiter.");
        }

        return recordDelimiterIndex;
    }
}

With this method, content decryption can be implemented:

public static class Aes128GcmEncoding
{
    private static async Task DecryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;

        byte[] cipherText = new byte[recordSize];
        byte[] plainText = null;
        int recordDataSize = recordSize - RECORD_OVERHEAD_SIZE;
        int recordDelimiterIndex = 0;

        do
        {
            int cipherTextLength = await source.ReadAsync(cipherText, 0, cipherText.Length);
            if (cipherTextLength == 0)
                throw new FormatException("Invalid records order or missing record(s).");

            aes128GcmCipher.Reset();
            AeadParameters aes128GcmParameters = new AeadParameters(new KeyParameter(contentEncryptionKey),
                128, GetNonce(pseudorandomKey, recordSequenceNumber));
            aes128GcmCipher.Init(false, aes128GcmParameters);

            byte[] plainText = new byte[aes128GcmCipher.GetOutputSize(cipherText.Length)];
            int lenght = aes128GcmCipher.ProcessBytes(cipherText, 0, cipherText.Length, plainText, 0);
            aes128GcmCipher.DoFinal(plainText, lenght);

            recordDelimiterIndex = GetRecordDelimiterIndex(plainText, recordDataSize);

            if ((plainText[recordDelimiterIndex] == LAST_RECORD_DELIMITER) && (source.ReadByte() != -1))
                throw new FormatException("Invalid records order or missing record(s).");

            await destination.WriteAsync(plainText, 0, recordDelimiterIndex);
        }
        while (plainText[recordDelimiterIndex] != LAST_RECORD_DELIMITER);
    }
}

HttpClient Plumbing

With the decoding implementation ready the components required by HttpClient can be prepared. I've decided to reuse the same wrapping pattern I used with Aes128GcmEncodedContent.

public sealed class Aes128GcmDecodedContent : HttpContent
{
    private readonly HttpContent _contentToBeDecrypted;
    private readonly Func<string, byte[]> _keyLocator;

    public Aes128GcmDecodedContent(HttpContent contentToBeDecrypted, Func<string, byte[]> keyLocator)
    {
        _contentToBeDecrypted = contentToBeDecrypted;
        _keyLocator = keyLocator;
    }

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        if (!_contentToBeDecrypted.Headers.ContentEncoding.Contains("aes128gcm"))
            throw new NotSupportedException($"Encryption type not supported or stream isn't encrypted.");

        Stream streamToBeDecrypted = await _contentToBeDecrypted.ReadAsStreamAsync();

        await Aes128GcmEncoding.DecodeAsync(streamToBeDecrypted, stream, _keyLocator);
    }

    protected override bool TryComputeLength(out long length)
    {
        length = 0;

        return false;
    }
}

But this time it is not our code which is creating the content object - it comes from the response. In order to wrap the content coming from the response, the HttpCLient pipeline needs to be extended with DelegatingHandler which will take care of that upon detecting the desired Content-Encoding header value. The DelegatingHandler also gives an opportunity for setting the Accept-Encoding header so the other side knows that encrypted content is supported.

public sealed class Aes128GcmEncodingHandler : DelegatingHandler
{
    private readonly Func<string, byte[]> _keyLocator;

    public Aes128GcmEncodingHandler(Func<string, byte[]> keyLocator)
    {
        _keyLocator = keyLocator;
    }

    protected override async Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("aes128gcm"));

        HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

        if (response.Content.Headers.ContentEncoding.Contains("aes128gcm"))
        {
            response.Content = new Aes128GcmDecodedContent(response.Content, _keyLocator);
        }

        return response;
    }
}

With those components in place, we can try requesting some encrypted content from the server.

Test Run

To see decryption in action, the HttpClient pipeline needs to be set to use the components created above (assuming the server will respond with encrypted content).

IDictionary<string, byte[]> _keys = new Dictionary<string, byte[]>
{
    { String.Empty, Convert.FromBase64String("yqdlZ+tYemfogSmv7Ws5PQ==") },
    { "a1", Convert.FromBase64String("BO3ZVPxUlnLORbVGMpbT1Q==") }
};
Func<string, byte[]> keyLocator = (keyId) => _keys[keyId ?? String.Empty];

HttpMessageHandler encryptedContentEncodingPipeline = new HttpClientHandler();
encryptedContentEncodingPipeline = new Aes128GcmEncodingHandler(keyLocator)
{
    InnerHandler = encryptedContentEncodingPipeline
};

using (HttpClient encryptedContentEncodingClient = new HttpClient(encryptedContentEncodingPipeline))
{
    string decryptedContent = encryptedContentEncodingClient.GetStringAsync("<URL>").Result;
}

This gives full support for aes128gcm content encoding in HttpClient. All the code is available here for anybody who would like to play with it.