Let's say your http_plugin.log contains something like this.

Missing, invalid, or expired certificate in certificate chain for server server1.example.com:12340 GSKVAL_ERROR_NO_CHAIN_BUILT (575010)

This probably means you have an IBM IHS HTTP server added to your WebSphere deployment manager and you are using the web server plugin (plugin-cfg.xml) and there is some SSL issue between your IBM IHS HTTP server added to your WebSphere deployment manager. Check out my article IBM WebSphere - Getting Started with the web server plugin (plugin-cfg.xml) for more details on the web server plugin (plugin-cfg.xml). 

For example, in the WebSphere admin console, at Servers > Server Types > Web servers > your IBM IHS HTTP web server > Plug-in properties, the path to the Key Database file on your IBM IHS HTTP web server should be displayed. In this example, the Key Database file is located at /path/to/plugin-key.kdb (which of course, is not a real path, but you get the idea). Check out my article IBM WebSphere - Setting web server plugin (plugin-cfg.xml) SSL for more details on this.

So first and foremost, I want to list the certificates in the plugin-key.kdb file. Check out my article IBM Global Security Kit (GSKit) - List certificates and private keys in a Key Database file for more details on this.

/path/to/gsk8/bin/gsk8capicmd_64 -cert -list -db /path/to/plugin-key.kdb -stashed

Something like this should be returned.

Certificates found
* default, - personal, ! trusted, # secret key
!       amazon
!       godaddy
!       verisign
-       default

The trusted certificates (beginning with the exclamation point) will be in the WebSphere admin console at Security > SSL certificate and key management > Key stores and certificates > CMSKeyStore > Signer certificates.

The personal certifiate (beginning with the dash) will be in the WebSphere admin console at Security > SSL certificate and key management > Key stores and certificates > CMSKeyStore > Personal certificates.

Check out my article IBM WebSphere - Personal certificate vs Signer certificate for more details on the difference between Personal certificates and Signer certificates.

Since the error in the http_plugin.log mentions that the certificate may be expired, I'll want to see if the default Personal certificate is expired. This can be done on the command line. Check out my article IBM Global Security Kit (GSKit) - Display certificate details. For example, when I was working through this issue, the default Personal certificate in my plugin-key.kdb file was indeed expired.

~]$ /path/to/gsk8capicmd_64 -cert -details -db /path/to/plugin-key.kdb -stashed -label default
Not Before : June 5, 2016 2:04:24 PM CDT
Not After : June 5, 2017 2:04:24 PM CDT

Or, you can issue the same command but use -validate instead of -details.

[webproc@dlmbweb011 ~]$ /path/to/gsk8capicmd_64 -cert -details -db /path/to/plugin-key.kdb -stashed -label default
CTGSK2048W The validity period does not include today or does not fall within its issuer's validity period.
Additional untranslated info:
GSKKM_LAST_VALIDATION_ERROR: GSKVAL_ERR_CERT_EXPIRED (575018)
GSKKM_VALIDATIONFAIL_SUBJECT: [Class=]GSKVALMethod::PKIX[Issuer=]CN=dmgr.example.com,OU=Root Certificate,OU=foo,OU=bar,O=IBM,C=US[#=]38b3dc1d1247[Subject=]CN=server1.example.com,OU=foo,OU=bar,O=IBM,C=US
CTGSK2048W The validity period does not include today or does not fall within its issuer's validity period.

Or, you can view the expiration date of the default Perosnal certificate in your CMSKeyStore using the WebSphere admin console.

Since the certificate was expired, I selected the default certificate and clicked on Renew to renew the certificate.

Or, you could renew the certificate using this wsadmin command.

AdminTask.renewCertificate('[-keyStoreName CMSKeyStore -keyStoreScope (cell):<the name of your cell>:(node):<the name of your node>:(server):<the name of your web server> -certificateAlias default ]')

Then in the WebSphere admin console at Servers > Server Types > Web servers > your IBM IHS HTTP web server > Plug-in properties click Copy to Web server key store directory.

Something like this should be returned.

Certificates found
* default, - personal, ! trusted, # secret key
!       amazon
!       godaddy
!       verisign
-       default

Then back on my IBM IHS HTTP web server, I reissued the following command and validated that the default Personal certifiate in the plugin-key.kdb was no longer expired. Nice!

~]$ /path/to/gsk8capicmd_64 -cert -details -db /path/to/plugin-key.kdb -stashed -label default
Not Before : June 5, 2016 2:04:24 PM CDT
Not After : June 5, 2017 2:04:24 PM CDT

If problems persist, then I would next look at the plugin-cfg.xml file and there should be a Transport section that contains the hostname and port being used in the SSL connection from your IBM IHS HTTP web server to your WebSphere deployment manager. Notice in this example that the port is 12345.

<Transport ConnectionTTL="28" Hostname="dmgr.example.com" HostnameAlias="dmgr.example.com" Port="12345" Protocol="https">
  <Property Name="keyring" Value="/opt/WebSphere/Plugins/config/mywebserver/plugin-key.kdb"/>
  <Property Name="stashfile" Value="/opt/WebSphere/Plugins/config/mywebserver/plugin-key.sth"/>
</Transport>

This should be the WC_defaulthost_secure port of one of your WebSphere application server.

dlmbpor-1-0001.thrivent.com:12340/portal/PA_Applications/

And I also saw this in http_plugin.log.

[23/May/2024:05:52:02.59150] 00004062 a4fd4740 - ERROR: Subject [
[Class=]GSKVALMethod::X509
[Issuer=]CN=my_root_ca,DC=foo,DC=com
[#=]480000048130f596eed49c0c0e000100000481
[Subject=]CN=was.example.com,O=Acme,L=LA,ST=California,C=US
[Class=]GSKVALMethod::PKIX
[Issuer=]CN=my_root_ca,DC=foo,DC=com
[#=]480000048130f596eed49c0c0e000100000481
[Subject=]CN=was.example.com,O=Acme,L=LA,ST=California,C=US] failed certificate validation

[23/May/2024:05:52:02.59151] 00004062 a4fd4740 - ERROR: X509 Certificate validation log: [
[Class=]GSKVALMethod::X509
[Time=]2024:5:23:5:52:2.590
[buildChain=]
[Error=]GSKVAL_ERR_NO_CHAIN_BUILT
[Info=]CN=my_root_ca,DC=foo,DC=com
[Cert=]
[Issuer=]CN=my_root_ca,DC=foo,DC=com
[#=]480000048130f596eed49c0c0e000100000481
[Subject=]CN=was.example.com,O=Acme,L=LA,ST=California,C=US
[=Cert][=buildChain]
[Class=]GSKVALMethod::PKIX
[Time=]2024:5:23:5:52:2.591
[buildChain=]
[Error=]GSKVAL_ERR_NO_CHAIN_BUILT
[Info=]CN=my_root_ca,DC=foo,DC=com
[Cert=]
[Issuer=]CN=my_root_ca,DC=foo,DC=com
[#=]480000048130f596eed49c0c0e000100000481
[Subject=]CN=was.example.com,O=Acme,L=LA,ST=Califorinia,C=US
[=Cert]
[=buildChain]

Notice in the output above that the CN of the server certificate is "was.example.com". When I looked through the certificates in the plugin-key.kdb, there was no certificate with CN was.example.com. So, that's why I was getting "Missing, invalid, or expired certificate in certificate chain".

So, what to do?

Back in the WebSphere admin console at Servers > Server Types > Web servers > your IBM IHS HTTP web server > Plug-in properties I clicked on the Manage keys and certificates button.

Which showed me that the plugin-key.kdb file was in the CMSKeyStore but the certificate chain containing the "was.example.com" certificate was in a different key store, the CellDefaultKeyStore.

When you create a new web server definition, WebSphere Application Server associates the web server plug-in with a Certificate Management Services (CMS) keystore for a specific node.

The keystore contains all of the signers for the current cell with the self-signed or chained certificate, which belongs to the node.

The plug-in can communicate securely to WebSphere Application Server, even when the plug-in is configured with Secure Sockets Layer (SSL) client authentication enabled

Before doing anything permanent, I just wanted to do some proof of concept, to see if adding the "was.example.com" certificate chain to the plugin-key.kdb file in the CMSKeyStore would resolve the problem. Fortunately, I was able to extract the root, intermediate, and was.example.com certificates from the key.p12 file on my Linux system using OpenSSL. For more details on this, check out my article OpenSSL - Extract certificate and private key from a PFX or P12 file. I used this command to view the key.p12 file.

openssl pkcs12 -in /tmp/key.p12 -info

And I then made three files.

• root.pem
• intermediate.pem
• was_example_com.pem

And I imported all three certificates into my plugin-key.kdb file. Check out my article IBM Global Security Kit (GSKit) - Add or Import a certificate into a Key Database file for more details on this.

/path/to/gsk8capicmd_64 -cert -add -file root.pem -label was_example_com_root -db /opt/WebSphere/Plugins/config/mywebserver/plugin-key.kdb -stashed

/path/to/gsk8capicmd_64 -cert -add -file intermediate.pem -label was_example_com_intermediate -db /opt/WebSphere/Plugins/config/mywebserver/plugin-key.kdb -stashed

/path/to/gsk8capicmd_64 -cert -add -file was_example_com.pem -label was_example_com -db /opt/WebSphere/Plugins/config/mywebserver/plugin-key.kdb -stashed

And I verified my plugin-key.kdb file contains the three imported certificates.

Certificates found
* default, - personal, ! trusted, # secret key
! was_example_com_root
! was_example_com_intermediate
! was_example_com_
-  my_root_ca
-  my_intermediate_certificate
*- my_server_certificate