Aug 21, 2016
Server-side SSL Failing
Hello everyone
We have a project where we need to provide TLS offload (client-side) and server-side encryption to the back-end web servers. I thought I'd test this out in my lab but seem to b...
The server's certificate is sent as part of a larger message, so the problem could be anything before it as well, including cipher mismatch.
Are you using a (subordinate) CA certificate and private key as the web server's cert/key? What are the keyUsage and extendedKeyUsage values in that certificate's X509 properties? You can see this from a Windows machine by double-clicking the certificate file of from OpenSSL on the command line:
openssl x509 -in [cert] -noout -text
Can you also provide an SSLDUMP of that server side handshake?
ssldump -AdNn -i [server side VLAN] port 443 [and any additional filters]
Can you also provide an SSLDUMP of the OpenSSL command line test?
Just spitballing here, but the error message seems to indicate there's something wrong with the server's public key. Even in lieu of validation, the server's public key can be used for encryption as part of the key exchange, so it might be worth a look at the handshake itself.