JSSE openssl SSLv3 bad_mac_record

JSSE openssl SSLv3 bad_mac_record

I recently ran into trouble with two of our client's sites that uses the WorldPay callback functionality. WorldPay uses the Java built in SSL/TLS library (JSSE).

The change I made to my sites when this started to happen was to upgrade our Apache web servers from the 1.3.x line to 2.0.x, but more importantly I upgraded from openssl 0.9.6b to 0.9.7d (because of a _bad_ mistake I thought I'd previously run 0.9.7c).

Shortly after this upgrade I started getting notifications about callbacks from WorldPay failing with a "bad_mac_record".

Further investigation seems to show that there is some incompatibility between JSSE and openssl 0.9.7d specifically when using SSLv3.

Here's my java test client: SSLTest.java

Running it against what I believe is a broken apache/mod_ssl/openssl.

$ java -cp . SSLTest testhost 443
javax.net.ssl.SSLException: Received fatal alert: bad_record_mac
at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.b(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.b(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(DashoA6275)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:334)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:402)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:406)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:150)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
at SSLTest.main(SSLTest.java:27)

Here is the apache error log when this happens.

At first I suspected that my build of apache2/openssl was flawed, however the above test was confirmed against an out-of-the-box apache2 build from the debian "sarge" distribution.

My relevant httpd.conf configuration is:

SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLSessionCache dbm:/var/log/apache2/ssl_scache
SSLSessionCacheTimeout 300
...
<VirtualHost>
...
SSLEngine on

# This 'SetEnvIf' is needed because of a bug in IE 4-6 where
# combination HTTP/1.1, Keep-Alive and SSL sometimes breaks.
# This forces the connection to be HTTP/1.0 for IE 4-6.
# ssl-unclean-shutdown means that the SSL socket connection is
# dropped rather than a notify message being sent to the client
# first as this notify message is part of the problem.

SetEnvIf User-Agent ".*MSIE (4|5|6).*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

SSLCertificateKeyFile ssl/testhost.key
SSLCertificateFile ssl/testhost.crt
SSLCertificateChainFile ssl/comodo-intermediate.crt

SSLProtocol SSLv3

# This 'SSLCipherSuite' is needed because of a bug in the SGC
# negotiation in IE4. If the digest algorithm (one of MD5, SHA
# or SHA1) differed between the 40-bit negotiation and the 128-bit
# renegotiation, then IE would break. The below disables all SHA
# and SHA1 related combinations, forcing MD5 to be choosen both times.

SSLCipherSuite !NULL:!SHA:!SHA1:RSA:ALL
...
</VirtualHost>

After some debugging I found that the problem could be fixed by changing:

SSLProtocol SSLv3

to

SSLProtocol All

This makes the JSSE choose TLSv1 rather than SSLv3 and the negotiation is successful.

Wednesday, 22 June 2005