Over a million developers have joined DZone.

Java Problem With Mutual TLS Authentication When Using Incoming/Outgoing Connections Simultaneously

DZone's Guide to

Java Problem With Mutual TLS Authentication When Using Incoming/Outgoing Connections Simultaneously

Mutual TLS authentication is a must in enterprise communication. We look at a scenario where it can be problematic when using Java, along with a couple of solutions.

· Security Zone ·
Free Resource

DON’T STRESS! Assess your OSS. Get your free code scanner from FlexeraFlexNet Code Aware scans Java, NuGet, and NPM packages.

In most enterprise environments, some form of secure communication (e.g. TLS or SSL) is used in connections between applications. In some environments, mutual (two-way) authentication is also a non-functional requirement. This is sometimes referred to as two-way SSL or mutual TLS authentication. So, as well as the server presenting its certificate it requests that the client send its certificate so that it can then be used to authenticate the caller.

A partner of my current client has been developing a server which receives data over MQTT and because the data is quite sensitive the customer decided that the data should be secured using mutual TLS authentication. Additionally, the customer requires that when the aggregated data which this server collects is posted to further downstream services, it is also done using mutual TLS authentication. This server needs to present a server certificate to its callers so that they can verify the hostname and identity, but, additionally, it must present a client certificate with a valid user ID to the downstream server when requested to do so during the SSL handshake.

The initial idea was to implement this using the standard JVM system properties for configuring a keystore: "-Djavax.net.ssl.keyStore=...", i.e. putting both client and server certificates into the single keystore. We soon realized, however, that this doesn't work, and tracing the SSL debug logs showed that the server was presenting the wrong certificate, either during the incoming SSL handshake or the outgoing SSL handshake. During the incoming handshake, it should present its server certificate. During the outgoing handshake, it should present its client certificate.

The following log extracts have been annotated and show the problems:

Upon further investigation, it became clear that the problem is related to the default key manager implementation in the JVM. The SunX509KeyManagerImpl class is used for selecting the certificate which the JVM should present during the handshake, and for both client certificate and server certificate selection, the code simply takes the first certificate it finds:

    String[] aliases = getXYZAliases(keyTypes[i], issuers);
    if ((aliases != null) && (aliases.length > 0)) {
        return aliases[0];  <========== NEEDS TO BE MORE SELECTIVE

The aliases returned by the method on the first line simply match key types (e.g. DSA) and optional issuers. So in the case where the keystore contains two or more certificates, this isn't selective enough. Furthermore, the order of the list is based on iterating over a HashMap entry set, so the order is not, say, alphabetical, but it is deterministic and constant. So while searching for the server certificate, the algorithm might return the client certificate. If, however, that part works, the algorithm will then fail when the server makes the downstream connection and needs to present its client certificate, as again the first certificate will be presented, namely the server certificate. As such, because it is impossible to create concurrent incoming and outgoing two-way SSL connections, I have filed a bug with Oracle (internal review ID 9052786 reported to Oracle on 20180225).

One solution is to use two keystores, one for each certificate as demonstrated here.

A possible patch for the JVM would be to make the algorithm more selective by using the"extended key usage" certificate extensions. Basically, the above code could be enhanced to additionally check the extended key usage and make a more informed decision during alias selection, for example:

String[] aliases = getXYZAliases(keyTypes[i], issuers);
if ((aliases != null) && (aliases.length > 0)) {
    String alias = selectAliasBasedOnExtendedKeyUsage(aliases, "");  //TODO replace with constant
    if (alias != null) return alias;

    //default as implemented in openjdk
    return aliases[0];

The method to select the alias would then be as follows:

private String selectAliasBasedOnExtendedKeyUsage(String[] aliases, String targetExtendedKeyUsage) {
    for(String alias : aliases){
        //assume cert in index 0 is the lowest one in the chain, and check its EKU
        X509Certificate certificate = this.credentialsMap.get(alias).certificates[0];

Try FlexNet Code Aware Today! A free scan tool for developers. Scan Java, NuGet, and NPM packages for open source security and license compliance issues.

security ,tls ,ssl ,enterprise security ,authentication

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}