How to Decrypt an AS2 Message (SMIME) With OpenSSL

I have been involved in the AdroitLogic AS2Gateway, a SaaS B2B AS2 messaging platform, for almost two years now, and one of the common issues we see in the users who are getting started with AS2 is decryption failure. In this blog post, we will look at what encryption and decryption are in AS2 protocol, how to decrypt an AS2 message, and some tips on figuring out the cause for certain decryption failures.

Encryption in AS2 Protocol

AS2 protocol basically uses public-key cryptography or asymmetric cryptography for encryption. There, the receiver's public key is used for encryption and receiver's private key is used for decryption as shown below.

Assuming the receiver's private key has not been compromised, encrypting data and messages offers the following security benefits.

• Confidentiality — Ensures that only the intended recipient can decrypt and view the contents, i.e. the content is encrypted with the recipient's public key, and hence, it can only be decrypted with the receiver's private key.
• Data Integrity — Determine whether the file or data the receiver got was altered along the way, i.e part of the decryption process involves verifying that the contents of the original encrypted message and the new decrypted match, so even the slightest change to the original content would cause the decryption process to fail.

Let's Get to Work!

For demonstration purposes, we will be using an incoming AS2 message to the AS2Gateway, and since we are only focusing on decryption in this blog post, the incoming AS2 message will is not signed or compressed.

Downloading RAW Message and Transport Headers

Once we have received an AS2 message, we can see the received message in the inbox view in AS2Gateway as shown below.

Then, we can click on the message subject (in this case, it is "Sample Encrypted Message") to go to the detailed view of the received message as shown below.

Now, you can click on the "Raw Message" button and "Download Transport Headers" button to download the unprocessed AS2 message payload and transport headers we received from the partner respectively. The raw message will be downloaded to a file with name message.raw, and the transport headers will be downloaded to a file with name headers.raw.

Getting the Receiver's Public and Private Key

Now that we have the raw message and transport headers, we need the receiver's public and private keys. As for the public key, you can directly download it by clicking the PEM (purple) button from the certificates view (shown below) in the AS2Gateway, and as for the private key, you will have to first download the JKS (identity.jks) by clicking on the JKS (red) button from the certificates view and extract the private key from the JKS. Check out my step-by-step guide on extracting the private key from JKS for more details.

Note that you'll need the key password and key store password when extracting the private key. If you do not remember them, you can view more details on the certificate by clicking on the common name (in this case, the common name on the relevant AS2 Station for this demonstration is "RJ_LOCAL"), and from there, you should be able to find the relevant passwords.

Before we proceed with the next steps, let's make sure we have everything we need in place.

• Raw message (message.raw)
• Transport headers (headers.raw)
• Receiver's private key (private_key.pem)
• Receiver's public key (cert.pem)

Analyzing the HTTP Transport Headers

Let's first take a look at the transport headers before we proceed.

date:Sun, 17 Mar 2019 09:06:25 IST
mime-version:1.0
subject:Sample Encrypted Message
as2-version:1.2
User-Agent:mendelson opensource AS2 1.1 build 51 - www.mendelson-e-c.com
recipient-address:http://service.as2gateway.org:8280/service/as2-receiver
ediint-features:multiple-attachments, CEM
as2-from:Mendelson_AS2ID
Expect:100-continue
content-disposition:attachment; filename="test_message.txt"
host:service.as2gateway.org:8280
message-id:
disposition-notification-to:http://localhost:8086/as2/HttpReceiver
content-type:application/pkcs7-mime; smime-type=enveloped-data; name=smime.p7m
connection:close, TE
from:sender@as2server.com
as2-to:RJ_LOCAL
disposition-notification-options:signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha1
Content-Length:411

As you can see, there are a bunch of headers, and thus, let us only focus on a couple of important ones in the context of decrypting the AS2 message.

• The content-type header suggests that we have an encrypted payload in the outer most layer.
• The content-disposition header gives away the file name of the payload to be test_message.txt.
• We also have the mime-version to be 1.0

If you are interested in knowing more in-depth details, the best place to start would be the AS2 RFC 4130.

So now, we know that the payload is encrypted (which should be the case since that is the type we selected for this demo) and we know that the file name is test_message.txt. Great. Now, we have almost everything we need to perform the decryption. Only a few more steps to go.

Encoding Raw Message in Base64

Since we are working with an encrypted raw message here, it is always better to convert it to base64 so that we can safely play with it using text editors. Of course, one can, and should, be able to proceed without converting it to base64 as well, but I prefer to convert the raw message to base64 for convenience in the next steps. Let's run the below command (here we use the 'base64' command line tool ) to covert the raw message to base64. Note that it is very important to have the parameter '-break=64,' which breaks the base64 output to lines with 64 characters, or else, you might run into an error during decryption.

base64 message.raw --break=64 > base64_message.raw

From now on, we will be working on following the base64_message.raw file.

MIAGCSqGSIb3DQEHA6CAMIACAQAxggECMIH/AgEAMGgwXjERMA8GA1UEAwwIUkpf
TE9DQUwxDTALBgNVBAoMBE5vbmUxDTALBgNVBAsMBE5vbmUxDTALBgNVBAcMBE5v
bmUxDTALBgNVBAgMBE5vbmUxDTALBgNVBAYTBE5vbmUCBgFieuE9LTANBgkqhkiG
9w0BAQEFAASBgCcPcpCqaSDJXn0F4C/wr69fQrAPLVRJgU2x5qNrz63jwl0jfGoq
RlR9/UTFol23SpqHx8TA1U3YBj0EDrGE9+qypXMAmAQB3GYCLFnbFDL3iv2ux+m8
6kzl2GttsMzli/fwtJhuVb8F8rdY4eNnvhcoSWEiJcLisHuKaGbKlNjpMIAGCSqG
SIb3DQEHATAUBggqhkiG9w0DBwQIyTHBk7MD9bOggARQI29+M12BFv/xqc1CHIg8
PIlCQfy0eSHRIsMAw3YjtBY4Vg2rgD0YrqHFF9lw8H3KgnG5FSytcTd/xQMQv99z
25x+IdJdn9G1tzmoRgVmhd0AAAAAAAAAAAAA

Adding Required Headers

Do you remember when we talked about a few important transport headers when we looking at the transport headers? Now is the time to use them. We need to add those headers to our base64_message.raw file so that the final output would be as follows. (Let's take the new file as base64_message_with_headers.raw) Note that the white space between the headers and the base64 encoded payload is intentional. You might notice that in addition to the headers we talked about earlier, we have added 'content-transfer-encoding: base64' to denote that content is in base64.

mime-version:1.0
content-disposition:attachment; filename="test_message.txt"
content-type:application/pkcs7-mime; smime-type=enveloped-data; name=smime.p7m
content-transfer-encoding: base64

MIAGCSqGSIb3DQEHA6CAMIACAQAxggECMIH/AgEAMGgwXjERMA8GA1UEAwwIUkpf
TE9DQUwxDTALBgNVBAoMBE5vbmUxDTALBgNVBAsMBE5vbmUxDTALBgNVBAcMBE5v
bmUxDTALBgNVBAgMBE5vbmUxDTALBgNVBAYTBE5vbmUCBgFieuE9LTANBgkqhkiG
9w0BAQEFAASBgCcPcpCqaSDJXn0F4C/wr69fQrAPLVRJgU2x5qNrz63jwl0jfGoq
RlR9/UTFol23SpqHx8TA1U3YBj0EDrGE9+qypXMAmAQB3GYCLFnbFDL3iv2ux+m8
6kzl2GttsMzli/fwtJhuVb8F8rdY4eNnvhcoSWEiJcLisHuKaGbKlNjpMIAGCSqG
SIb3DQEHATAUBggqhkiG9w0DBwQIyTHBk7MD9bOggARQI29+M12BFv/xqc1CHIg8
PIlCQfy0eSHRIsMAw3YjtBY4Vg2rgD0YrqHFF9lw8H3KgnG5FSytcTd/xQMQv99z
25x+IdJdn9G1tzmoRgVmhd0AAAAAAAAAAAAA

Decrypting...

It's time to run the decryption command. Here, we use the 'smime' tool by OpenSSL.

openssl smime -decrypt -in base64_message_with_headers.raw -recip cert.pem -inkey private_key.pem >> test_message.txt

Once you run the command, you should have the output in the test_message.txt file. Note that, in this case, we will get the plain text output since we used a payload without compression and signing.

This is a test message for the demonstration of AS2 decryption by OpenSSL.

As I stated before, if the base64 output is not split into lines with 64 characters, you may get an error similar to following. I thought of adding that for completion so that whoever else faces that issue may find the solution here.

Error reading S/MIME message
4545414764:error:0DFFF07B:asn1 encoding routines:CRYPTO_internal:header too long:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.240.1/libressl-2.6/crypto/asn1/asn1_lib.c:152:
4545414764:error:0DFFF06E:asn1 encoding routines:CRYPTO_internal:decode error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.240.1/libressl-2.6/crypto/asn1/asn_mime.c:195:
4545414764:error:0DFFF0CB:asn1 encoding routines:CRYPTO_internal:asn1 parse error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.240.1/libressl-2.6/crypto/asn1/asn_mime.c:525:

That concludes the steps on decrypting the payload. Even though we've looked at doing the decryption entirely, using command line tools in this article, this can be done using a few lines on Java code as well. I hope to cover it in a future article.

Bonus Pack

Before signing off, I would like to share some bonus details that would help you identify the cause for certain decryption failure scenarios. The first one is on how to find out the encryption algorithm used.

Finding Out the Encryption Algorithm Used

In order to find the encryption algorithm used, we can use the asn1parse tool by OpenSSL. Let us run the following command to get the asn1parse output. (Note that if you run the command without the  -inform der parameter you might get an error as 'Error: offset too large')

openssl asn1parse -inform der -in message.raw

The output would be as follows. If you can see below, there are roughly two main parts shown here in ASN.1 notation as  pkcs7-envelopedData part and pkcs7-data part. In pkcs7-data part, we have  des-ede3-cbc, which is the encryption algorithm used.

    0:d=0  hl=2 l=inf  cons: SEQUENCE          
    2:d=1  hl=2 l=   9 prim: OBJECT            :pkcs7-envelopedData
   13:d=1  hl=2 l=inf  cons: cont [ 0 ]        
   15:d=2  hl=2 l=inf  cons: SEQUENCE          
   17:d=3  hl=2 l=   1 prim: INTEGER           :00
   20:d=3  hl=4 l= 258 cons: SET               
   24:d=4  hl=3 l= 255 cons: SEQUENCE          
   27:d=5  hl=2 l=   1 prim: INTEGER           :00
   30:d=5  hl=2 l= 104 cons: SEQUENCE          
   32:d=6  hl=2 l=  94 cons: SEQUENCE          
   34:d=7  hl=2 l=  17 cons: SET               
   36:d=8  hl=2 l=  15 cons: SEQUENCE          
   38:d=9  hl=2 l=   3 prim: OBJECT            :commonName
   43:d=9  hl=2 l=   8 prim: UTF8STRING        :RJ_LOCAL
   53:d=7  hl=2 l=  13 cons: SET               
   55:d=8  hl=2 l=  11 cons: SEQUENCE          
   57:d=9  hl=2 l=   3 prim: OBJECT            :organizationName
   62:d=9  hl=2 l=   4 prim: UTF8STRING        :None
   68:d=7  hl=2 l=  13 cons: SET               
   70:d=8  hl=2 l=  11 cons: SEQUENCE          
   72:d=9  hl=2 l=   3 prim: OBJECT            :organizationalUnitName
   77:d=9  hl=2 l=   4 prim: UTF8STRING        :None
   83:d=7  hl=2 l=  13 cons: SET               
   85:d=8  hl=2 l=  11 cons: SEQUENCE          
   87:d=9  hl=2 l=   3 prim: OBJECT            :localityName
   92:d=9  hl=2 l=   4 prim: UTF8STRING        :None
   98:d=7  hl=2 l=  13 cons: SET               
  100:d=8  hl=2 l=  11 cons: SEQUENCE          
  102:d=9  hl=2 l=   3 prim: OBJECT            :stateOrProvinceName
  107:d=9  hl=2 l=   4 prim: UTF8STRING        :None
  113:d=7  hl=2 l=  13 cons: SET               
  115:d=8  hl=2 l=  11 cons: SEQUENCE          
  117:d=9  hl=2 l=   3 prim: OBJECT            :countryName
  122:d=9  hl=2 l=   4 prim: PRINTABLESTRING   :None
  128:d=6  hl=2 l=   6 prim: INTEGER           :01627AE13D2D
  136:d=5  hl=2 l=  13 cons: SEQUENCE          
  138:d=6  hl=2 l=   9 prim: OBJECT            :rsaEncryption
  149:d=6  hl=2 l=   0 prim: NULL              
  151:d=5  hl=3 l= 128 prim: OCTET STRING      [HEX DUMP]:270F7290AA6920C95E7D05E02FF0AFAF5F42B00F2D5449814DB1E6A36BCFADE3C25D237C6A2A46547DFD44C5A25DB74A9A87C7C4C0D54DD8063D040EB184F7EAB2A57300980401DC66022C59DB1432F78AFDAEC7E9BCEA4CE5D86B6DB0CCE58BF7F0B4986E55BF05F2B758E1E367BE172849612225C2E2B07B8A6866CA94D8E9
  282:d=3  hl=2 l=inf  cons: SEQUENCE          
  284:d=4  hl=2 l=   9 prim: OBJECT            :pkcs7-data
  295:d=4  hl=2 l=  20 cons: SEQUENCE          
  297:d=5  hl=2 l=   8 prim: OBJECT            :des-ede3-cbc
  307:d=5  hl=2 l=   8 prim: OCTET STRING      [HEX DUMP]:C931C193B303F5B3
  317:d=4  hl=2 l=inf  cons: cont [ 0 ]        
  319:d=5  hl=2 l=  80 prim: OCTET STRING      [HEX DUMP]:236F7E335D8116FFF1A9CD421C883C3C894241FCB47921D122C300C37623B41638560DAB803D18AEA1C517D970F07DCA8271B9152CAD71377FC50310BFDF73DB9C7E21D25D9FD1B5B739A846056685DD
  401:d=5  hl=2 l=   0 prim: EOC               
  403:d=4  hl=2 l=   0 prim: EOC               
  405:d=3  hl=2 l=   0 prim: EOC               
  407:d=2  hl=2 l=   0 prim: EOC               
  409:d=1  hl=2 l=   0 prim: EOC

Finding Out the Public Key Used for Encryption

Bonus package is not done yet. Sometimes, it is important to figure out the certificate used in encryption to make sure if the sender has used the correct public key of the recipient. If you look at the asn1parse output above, you should see that we have commonName, organizationName, etc. in pkcs7-envelopedDatasection. These are the details on the certificate used to encrypt the AS2 payload.

Now, after the  countryName entry, you might see a line as follows.

122:d=9 hl=2 l= 4 prim: PRINTABLESTRING :None
128:d=6 hl=2 l= 6 prim: INTEGER :01627AE13D2D

This is the certificate serial in hex, and with this, you can verify if the correct public has been used during the encryption.

That concludes the bonus pack. May all your AS2 decryption failures go away!