Hello,I'm running httpd 2.2.3 on RHEL (2.6.18), trying to get client certificate authentication working on the last node (distributed setup using F5 GTM), but running into issues where the last httpd server starts to accept client certificates (browsers prompt for certificate) for about 10min, but then suddenly stops prompting. It gets even weirder (detailed below). Every other httpd instance works great.I have a server cert issues to by our CA, set it up on httpd A and exported it and running in on httpd B. Traffic is redirected by the F5 GTM based on routing times. This has worked great for several years. Now we're adding client certificate authentication where the client certificates are issues by the same CA. We want the authentication to be optional for now while we're transitioning, so if the client has not yet received their certificate they can still gracefully degrade back to the form-based authentication. This works great on our dev server, our canary server, and on one of the production servers (httpd A). When setting the same configuration on httpd B, though, something really strange starts to happen. At first, clients were not bring prompted for their certificate choice in their browser from httpd B. In troubleshooting, I changes the SSLVerifyClient option from 'optional' to 'require', in hopes to verify my client certificate chain file and the like. Doing this causes httpd B to request the client certificate (as seen in the browser). With this setting on 'require', the httpd B server continues to work as expected. I then change this setting to 'optional', stop the httpd service (check and ensure all httpd processes have been killed) and restart the httpd service. I check again with my browser - it works! To summarize, the only thing I did was change the SSLVerifyClient option from 'optional' to 'require' and the back again - no other changes. Confused, by triumphant, I re-enable the site on the F5. Minutes later, users are no longer prompted for their CA certs on that node - they are going directly to the form. I fail the node, and start troubleshooting again.After hours of reading logs (set SSL logging to 'info', nothing really significant here) and verifying SSL settings - I'm starting to pull my hair out and hoping to get some "fresh eyes" on the situation. Reloading or restarting the httpd service does not fix it (I tried doing this a dozen or so times, really without any empirical test reasoning other than to prove that it wasn't fluke with the 'require'/'optional' flipping. I then did the flip - 'optional' to 'require' and verified it worked again. Then, flipped it back 'require' to 'optional'...and again, it still works for about 10 min. The only thing I could think of is that httpd B is making AJAX calls to the DNS entry, which may be redirected back to httpd A, causing issues on the client somehow (we're LB at the OSI 3 level, so both servers have the same key pair). But I switched between different clients (RHEL/Windows) just going to httpd b and no love. Not until I restart the httpd service after changing it to 'require', then restart it again after changing it back to 'optional' does it work for a bit!All servers are running the same RHEL and Apache versions (they are VM templates of each other). httpd is configured identically, as confirmed by a 'diff'. They are running the same certs - and httpd B works when on 'require'.I was very happy with how easy this development solution and administrative transition was going - until the last server =).Other than Murphy's law, any ideas on what could be going on?Thanks,S
|