How to Analyze Java SSL Errors

In my recent projects I've had to do a lot with certificates, java and HTTPS with client-side authentication. In most of these projects, either during testing, or setting up a new environment, I've run into various SSL configuration errors that often resulted in a rather uncomprehensive error such as:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) 

In most of the cases it was misconfiguration where keystores didn't containt the correct certificates, the certificate chain was incomplete or the client didn't supply a valid certificate. So in the last project I decided to document what was happening and what caused specific errors during the SSL handshake.

In this article I'll show you why specific SSL errors occur, how you can detect them by analyzing the handshake information, and how to solve them. For this I use the following scenario:

• Server uses a certificate issued by a CA and requires client authentication. The server uses a simple truststore that lists this CA as trusted.
• Client connects using a certificate issued by this single trusted CA and has it's own trustore that also contains this certificate from the server.

Not a very complicated situation, but one you often see. Note that the following information can also be used to identify problems when you don't work with client certificates or use self-signed certificates. The way to determine the problem in those cases, is pretty much the same.

Happy Flow

First we'll look at the happy flow, what happens in the handshake when we use client certificates. We won't look at the complete negotiation phase, but only until both the client and the server have exchanged their certificates and have validated the received certificate. If everything goes well until that point, the rest should work. The following is what you see when you run the client and the server using the java VM parameter: -Djavax.net.debug=ssl:handshake.

The first thing that happens is that the client sends a ClientHello message using the TLS protocol version he supports, a random number and a list of suggested cipher suites and compression methods. From our client this looks like this:

Client sends:

*** ClientHello, TLSv1
RandomCookie:  GMT: 1331663143 bytes = { 141, 219, 18, 140, 148, 60, 33, 241, 10, 21, 31, 90, 88, 145, 34, 153, 238, 105, 148, 72, 163, 210, 233, 49, 99, 224, 226, 64 }
Session ID:  {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
***

The server responds, very originally, with a ServerHello message, that contains the choices made based on the information provided by the client another random number and (optionally) a session id.

Server sends:

*** ServerHello, TLSv1
RandomCookie:  GMT: 1331663143 bytes = { 172, 233, 79, 197, 14, 21, 187, 161, 114, 206, 7, 38, 188, 228, 120, 102, 115, 214, 155, 86, 211, 41, 156, 179, 138, 2, 230, 81 }
Session ID:  {79, 96, 145, 39, 203, 136, 206, 69, 170, 46, 194, 17, 154, 175, 13, 138, 143, 199, 162, 193, 110, 86, 113, 109, 248, 187, 220, 169, 47, 180, 44, 68}
Cipher Suite: SSL_RSA_WITH_RC4_128_MD5
Compression Method: 0
Extension renegotiation_info, renegotiated_connection: <empty>
***

So in this case we're going to use SSL_RSA_WITH_RC4_128_MD5 as Cipher Suite. The next step is also done by the server. The server next sends a Certificate message that contains its complete certificate chain:

Server sends:

*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=server, C=NL
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 143864428144045085986129639694300995179398936575198896494655652087658861594939489453166811774109137006267822033915476680673848164790815913192075840268069822357600376998775923266017630332239546722181180383155088413406178660120548292599278819762883993031950564327152510982887716901499177102158407884939613382007
  public exponent: 65537
  Validity: [From: Wed Mar 14 13:32:04 CET 2012,
               To: Thu Mar 14 13:32:04 CET 2013]
  Issuer: CN=Application CA, OU=GKD, O=Smartjava, L=Maasland, ST=ZH, C=NL
  SerialNumber: [    a881d144 5e631f21]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: C3 56 81 7F 33 91 8A FF   84 5E 0B BA 7A 01 D8 41  .V..3....^..z..A
0010: 6B 47 B2 F7 8F FB B5 77   23 D8 FB B2 35 19 6E C4  kG.....w#...5.n.
0020: A4 6A BC 23 BB 69 92 F6   85 5A 1E CB FE 23 C6 98  .j.#.i...Z...#..
0030: A0 57 F8 FB E9 DB B0 40   BD 8E F8 35 F8 77 E1 09  .W.....@...5.w..
0040: 5A 2E 45 71 80 F6 89 E7   0B 93 E2 48 EB 40 92 13  Z.Eq.......H.@..
0050: 14 AA 1F 59 AA 98 67 46   9B 52 33 49 9A 3C 91 9B  ...Y..gF.R3I.<..
0060: F1 CB 8A BD 7D D4 DD 76   C4 15 00 36 A3 B2 87 A7  .......v...6....
0070: D5 FF 52 E3 68 D4 F0 E0   32 86 74 02 DD 92 EC 1D  ..R.h...2.t.....

]
chain [1] = [
[
  Version: V3
  Subject: CN=Application CA, OU=SL, O=SmartJava, L=Waalwijk, ST=ZH, C=NL
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 159927271510058538658170959055540487654246676457579822126433656091883150307639380685203152841988861440546492270915750324654620063428634486478674507234742748515614639629692189315918046446256610037776978028900716455223387878926383828815082154427031884246429239077082613371662803582187768145965112751392402313823
  public exponent: 65537
  Validity: [From: Mon Mar 12 13:35:16 CET 2012,
               To: Wed Apr 11 14:35:16 CEST 2012]
  Issuer: CN=Application CA, OU=CA, O=Blaat, L=Waalwijk, ST=ZH, C=NL
  SerialNumber: [    fe7636c5 6804e69c]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 6C CC 48 03 E4 BE 07 D6   9E F6 4C 78 53 54 A2 B8  l.H.......LxST..
0010: 7B DA 40 09                                        ..@.
]
]

[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 6C CC 48 03 E4 BE 07 D6   9E F6 4C 78 53 54 A2 B8  l.H.......LxST..
0010: 7B DA 40 09                                        ..@.
]

]

[3]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 1A 30 08 15 01 8E A6 36   5F 38 22 C6 81 5E 69 B1  .0.....6_8"..^i.
0010: 42 9A 1E FF 0F C4 D7 40   5F 85 0E 42 35 E0 CC 00  B......@_..B5...
0020: 6E A5 2E 70 6B 79 64 C5   99 AE A4 29 CB 26 DE 60  n..pkyd....).&.`
0030: 0B A6 AB 19 06 6F 19 54   6C 1A 88 9E 3A 6A D4 BB  .....o.Tl...:j..
0040: CB 28 85 2F 72 4D DE 35   C0 9B F4 2F EF 8E 6D E8  .(./rM.5.../..m.
0050: 30 AC 12 7D B4 0D A3 08   DA D4 60 46 94 BD 12 AF  0.........`F....
0060: 44 F7 C3 B8 9D 69 2D 6A   32 C8 4D AE 12 60 05 09  D....i-j2.M..`..
0070: FE AE D0 1A 72 6D 91 CE   DA 7C 8E D5 31 14 31 4C  ....rm......1.1L

]

In this message you can see that the issuer of this certificate is our example CA. Our client checks to see if this certificate is trusted, which it is in this case. Since we require the client to authenticate itself the server requests a certificate from the client and after that sends a helloDone.

Server sends:

*** CertificateRequest
Cert Types: RSA, DSS
Cert Authorities:
<CN=Application CA, OU=CA, O=Blaat, L=Waalwijk, ST=ZH, C=NL>
*** ServerHelloDone

In this message you can see that the server provides a list of Cert Authorities it trusts. The client will use this information to determine if it has a keypair that matches this CA. In our happy flow, it has one and responds with a Certificate message.

Client sends:

*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=Application 3, OU=Smartjava, O=Smartjava, L=NL, ST=ZH, C=NL
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 90655907749318585147523875906892969031300830816947226352221659107570169820452561428696751943383590982109524990627182456571533992582229229163232831159652561902456847954385746762477844009336466314872376131553489447601649924116778337873632641536164462534398137791450495316700015095054427027256393580022887087767
  public exponent: 65537
  Validity: [From: Mon Mar 12 15:13:24 CET 2012,
               To: Tue Mar 12 15:13:24 CET 2013]
  Issuer: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL
  SerialNumber: [    b247ffb2 ce060768]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 97 58 36 C5 28 87 B3 16   9B DD 31 0C E0 C6 23 76  .X6.(.....1...#v
0010: 72 82 5B 13 4D 23 B6 0E   A9 2F 9F 0C 3F 97 15 6E  r.[.M#.../..?..n
0020: 7B 38 EC DE E2 57 D7 AA   07 12 E3 98 B7 86 A7 CE  .8...W..........
0030: 57 8E A1 29 96 C9 F0 30   57 67 C7 F1 F2 98 90 64  W..)...0Wg.....d
0040: 6C B9 6C 05 24 8B 56 3F   B1 FF 03 62 3D 81 DB 45  l.l.$.V?...b=..E
0050: D3 1F C1 B2 DD 77 CF 74   54 EB 9D 82 23 89 1A 70  .....w.tT...#..p
0060: F8 C4 68 6A B7 41 C7 DE   7B B6 3A 0C 17 E7 FA 98  ..hj.A....:.....
0070: 19 0C D8 91 FB 5E FE D2   B3 92 FD 2D 2A 6B 51 10  .....^.....-*kQ.

]
chain [1] = [
[
  Version: V3
  Subject: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 159927271510058538658170959055540487654246676457579822126433656091883150307639380685203152841988861440546492270915750324654620063428634486478674507234742748515614639629692189315918046446256610037776978028900716455223387878926383828815082154427031884246429239077082613371662803582187768145965112751392402313823
  public exponent: 65537
  Validity: [From: Mon Mar 12 13:35:16 CET 2012,
               To: Wed Apr 11 14:35:16 CEST 2012]
  Issuer: CN=Application CA, OU=Smartjava, O=Smartjava, L=Maasland, ST=ZH, C=NL
  SerialNumber: [    fe7636c5 6804e69c]

Certificate Extensions: 3
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 6C CC 48 03 E4 BE 07 D6   9E F6 4C 78 53 54 A2 B8  l.H.......LxST..
0010: 7B DA 40 09                                        ..@.
]
]

[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 6C CC 48 03 E4 BE 07 D6   9E F6 4C 78 53 54 A2 B8  l.H.......LxST..
0010: 7B DA 40 09                                        ..@.
]

]

[3]: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 1A 30 08 15 01 8E A6 36   5F 38 22 C6 81 5E 69 B1  .0.....6_8"..^i.
0010: 42 9A 1E FF 0F C4 D7 40   5F 85 0E 42 35 E0 CC 00  B......@_..B5...
0020: 6E A5 2E 70 6B 79 64 C5   99 AE A4 29 CB 26 DE 60  n..pkyd....).&.`
0030: 0B A6 AB 19 06 6F 19 54   6C 1A 88 9E 3A 6A D4 BB  .....o.Tl...:j..
0040: CB 28 85 2F 72 4D DE 35   C0 9B F4 2F EF 8E 6D E8  .(./rM.5.../..m.
0050: 30 AC 12 7D B4 0D A3 08   DA D4 60 46 94 BD 12 AF  0.........`F....
0060: 44 F7 C3 B8 9D 69 2D 6A   32 C8 4D AE 12 60 05 09  D....i-j2.M..`..
0070: FE AE D0 1A 72 6D 91 CE   DA 7C 8E D5 31 14 31 4C  ....rm......1.1L

]

This certificate is checked on the server side and if all is well, the final steps in the handshake are executed to setup the secured connection. Note that there is a CertificateVerify step. In this step the client signs a message with its private key. This is done so the server can verify the client has access to its private key. This might seem a step where things can go wrong in an incorrectly configured environment. In the default java implementation this won't happen. In the phase where the client has to determine which certificate to present to the server, the java implementation already checks if the privatekey is available.

What could possibly go wrong

So what could possibly go wrong in this handshake? In the next couple of sections we'll look at some scenarios, and how to detect them.

Passwords

Now that we've seen what happens when things go right, lets look at a couple of scenarios where things go wrong. We'll start simple with the following exception, that we get at the moment we start up the client application:

Exception in thread "main" java.security.UnrecoverableKeyException: Cannot recover key
at sun.security.provider.KeyProtector.recover(KeyProtector.java:311)
at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:121)
at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:38)
at java.security.KeyStore.getKey(KeyStore.java:763)
at com.sun.net.ssl.internal.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:113)
at com.sun.net.ssl.internal.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:48)
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:239)
at org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:186)
at org.apache.http.conn.ssl.SSLSocketFactory.(SSLSocketFactory.java:260)

This very helpful message is thrown when (from the javadoc) " .. a key in the keystore cannot be recovered". There are a couple of reasons this can happen, but normally this occurs when the key in the keystore is accessed with the wrong password. Usually when you use the keytool to create and manage your keys, the keystore password is usually the same as the key password. However, if you import keys from a PKCS#12 type keystore, the password of the keystore can be easily set to a different value. Not all the SSL client allow you to specify a different password for the key and the keystore. If that is the case you can use the following command, to change the password of the key:

keytool -keypasswd -alias <keyalias> -keystore <keystore>

It is also possible to set an incorrect password for the keystore. Luckily in that case the error message that is thrown is much more helpful:

Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:771)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38)
at java.security.KeyStore.load(KeyStore.java:1185)
    ...
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769)
... 3 more

If this occurs at the server side, we can see the same message when the SSL listener is being set up.

Incomplete CA Chains

Now lets look at the first of the "peer not authenticated" exceptions. In the logging we see this exception at the client side:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)

So enable SSL logging, run again, and we'll start with analyzing the handshake. We'll start by looking from the client side. If we look through the logging we find the following CertificateRequest message from the server and the ServerHelloDone.

*** CertificateRequest
Cert Types: RSA, DSS
Cert Authorities:
*** ServerHelloDone

So thus far, everything went ok. The server has already sent its certificate, and since our client doesn't throw an error on that part, we can assume it is trusted by the client. So something seems to be wrong with the steps that come after this message from the server.
If you look closer at this message, you can see that the server doesn't specify a set of Cert Authorities it trusts. This could be a misconfiguration at the server side, or it could just be that the server expects one of the trusted Root CAs. In any case, the client is free to send any certificate he wants. So the client sends the following certificate:

*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL
  Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3

...
]
chain [1] = [
[
  Version: V3
  Subject: EMAILADDRESS=jos.dirksen@gmail.com, CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL
  Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3
  ...
]

According to the specification the client now continues with the key exchange and generates secrets to exchange. Somewhere along the lines we can see the following:

pool-1-thread-1, WRITE: TLSv1 Handshake, length = 32
pool-1-thread-1, READ: TLSv1 Alert, length = 2
pool-1-thread-1, RECV TLSv1 ALERT:  fatal, internal_error
pool-1-thread-1, called closeSocket()

This means we've received an internal error. So something at the server side went wrong. Looking at the server we see the following in the SSL dump:

*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL
  Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3
  ...
]
chain [1] = [
[
  Version: V3
  Subject: EMAILADDRESS=jos.dirksen@gmail.com, CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL
  Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3
  ...
]
***
qtp1735121130-17, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
qtp1735121130-17, SEND TLSv1 ALERT:  fatal, description = internal_error
qtp1735121130-17, WRITE: TLSv1 Alert, length = 2

You can see that we received the certificate from the client, and directly after that we get this error. This error however doesn't really tell us anything. We do however have enough information to at least limit the possible errors. We know that the server didn't sent a list of CAs, we can see that the client sent a valid certificate, and that server somehow isn't able to process it. It looks like a problem with the server truststore. In this case the best approach is to look at the certificates the server trusts. Either in the cacerts file or in it's own truststore. Validate whether the CA certificate our client sends is in the server's truststore, and the server actually loads the stores we expect.

It's of course also possible that the client has an incomplete chain of trust for the certificate received from the server. In that case we once again get the "peer not authenticated" error at the client side. If we look at the SSL debug logging, we see the following exception occuring at the client side:

pool-1-thread-1, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
pool-1-thread-1, SEND TLSv1 ALERT:  fatal, description = internal_error
pool-1-thread-1, WRITE: TLSv1 Alert, length = 2

This exception occured directly after the server has sent its certificate using a "Certificate message":

*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=server, C=NL
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

Following the same reasoning as for the server we can conclude that there is something wrong with the client side truststore. For completeness sake, the server receives this error message when this situation occurs at the client:

qtp1500389297-17, READ: TLSv1 Alert, length = 2
qtp1500389297-17, RECV TLSv1 ALERT:  fatal, internal_error
qtp1500389297-17, called closeSocket()
qtp1500389297-17, handling exception: javax.net.ssl.SSLException: Received fatal alert: internal_error
qtp1500389297-17, called close()
qtp1500389297-17, called closeInternal(true)

Invalid keys

For the next exercise lets look at the following error that occurs during this handshake. In the logging at the client side we see the following error message in the SSL output:

ool-1-thread-1, WRITE: TLSv1 Handshake, length = 32
pool-1-thread-1, READ: TLSv1 Alert, length = 2
pool-1-thread-1, RECV TLSv1 ALERT:  fatal, internal_error
pool-1-thread-1, called closeSocket()
pool-1-thread-1, handling exception: javax.net.ssl.SSLException: Received fatal alert: internal_error
pool-1-thread-1, IOException in getSession():  javax.net.ssl.SSLException: Received fatal alert: internal_error

Which results in the very unhelpful:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)

When you receive an internal error, there is usually something wrong at the server side. So looking at the serverside, lets see what caused this error.

***
qtp2044601711-16, handling exception: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
qtp2044601711-16, SEND TLSv1 ALERT:  fatal, description = internal_error

Hmm.. somewhat more useful. It seems that there is something wrong with the algorithm we used, the client seems to have provided an incorrect certificate. But what is wrong? If you look back at the happy flow, you can send that at a certain time the server asks the client for a certificate using a "Certificate" message. Lets look a bit closer at this message and the response:

*** CertificateRequest
Cert Types: RSA, DSS
Cert Authorities:
<EMAILADDRESS=jos.dirksen@gmail.com, CN=CA2, OU=Smartjava, O=Smartjava, L=Waalwijk, ST=NB, C=NL>
*** ServerHelloDone
matching alias: application4
*** Certificate chain
chain [0] = [
[
  Version: V1
  Subject: CN=Application4, OU=Smartjava, O=Smartjava, L=NL, ST=NB, C=NL
  Signature Algorithm: SHA1withDSA, OID = 1.2.840.10040.4.3

  Key:  Sun DSA Public Key
  ...

What you can see here is that the server specifies the cert types it accepts, and the authorities it accepts. The client responses in this case however with a DSA public key. Depending on the server implementation this can cause this strange message. Another possible scenario I've seen (especially with self-signed certificates) is that with a "CertificateRequest" message like this:

*** CertificateRequest
Cert Types: RSA, DSS
Cert Authorities:
*** ServerHelloDone

This client won't respond with a certificate at all, if you only have DSA based keys in your keystore. It won't throw an error on the client side, but will cause a "null certificate chain" message as the server side. I haven't seen this scenario, though, when you don't use self-signed certificates.

Certificate expiration

So far we've seen how you can analyze the SSL handshake to determine where to look for configuration errors. In this last example we'll look at what happens when a certificate expires. In this case we once again see the very cryptic message at the client side:

pool-1-thread-1, READ: TLSv1 Alert, length = 2
pool-1-thread-1, RECV TLSv1 ALERT:  fatal, certificate_unknown
pool-1-thread-1, called closeSocket()
pool-1-thread-1, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
pool-1-thread-1, IOException in getSession():  javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
pool-1-thread-1, called close()
pool-1-thread-1, called closeInternal(true)
pool-1-thread-1, called close()
pool-1-thread-1, called closeInternal(true)
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)

If we look at the phase of the SSL handshake we're in, we can see that we've already sent our client certificate and finishing up the handshake when we receive this error. The error on the serverside is actually pretty helpful. After receiving the invalid certificate, in the debug logging, it shows us the following:

***
qtp1735121130-17, SEND TLSv1 ALERT:  fatal, description = certificate_unknown
qtp1735121130-17, WRITE: TLSv1 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 01 00 02 02 2E                               .......
qtp1735121130-17, called closeSocket()
qtp1735121130-17, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
qtp1735121130-17, called close()
qtp1735121130-17, called closeInternal(true)

It tells us that during the validation of the certificate, a timestamp check failed. This tells us that we should look at the validity of the certificates in our certificate chain to see what is happening.

Summary

In this article you've seen a couple of common causes for SSL exceptions and ways to identify the exception. Their can be many causes for these kind of exceptions, the most common though are the following:

• Incorrect certificate chains in the client truststore
• Incorrect certificate chains in the server truststore
• Invalid key algorithm used for private keys
• Expired certificate or expired CA certificate
• Incorrect passwords used to access the keys
• Multiple private keys to choose from

If you're presented with a such an exception a good general approach is this. You first check the keystores that are involved. Use the java keytool for this:

keytool -list -v -keystore <location_of_keystore>

This will print out all the certificates and keys in the keystore. Check whether the keys are of a supported type, the required CA certificates are stored and that your application is using the correct one (spent hours figuring out an issue because I was looking into a truststore for my private key). If everything seems to be OK at first glance it's time to enable ssl debugging (-Djavax.net.debug=ssl:handshake) and check the handshake messages that are sent. Wikipedia has a nice overview of which message is sent at a specific time. For more information on the content of the messages look at the RFC 5246 (or the one of the SSL/TLS version you're using, but the handshake changes are minimal between versions). Using the messages and the handshake, determine at what place in the handshake things go wrong, taking into account that the client will continue with the handshake, while the server is processing it's certificate.