CheckTLS.com gives me a clean bill of health both sending and receiving.

The following give me a clean output as well:
  openssl s_client -connect smtp.vintners.net:25 -starttls smtp
  openssl verify -CAfile SSL_WILDCARD_CertificateAuthorityRoot.crt -untrusted ov_chain.crt STAR.VINTNERS.NET.crt
Different renditions of this all seem to give a "self-signed certificate in certificate chain" for the first entry. I believe this is expected as this is the beginning of the world. This is not the error I'm seeing in the log

So I think I'm barking up the wrong tree. Let's try one of the servers that's giving me trouble:
  openssl s_client -cert STAR.VINTNERS.NET.crt -key star.vintners.net.key -CAfile ov_chain.crt -connect gmail.com:443
Yes! This is at least giving me the same error as I'm seeing in the logs, so maybe this is what I need to work on.

I changed something, and managed to break things so that suddenly every connection to the big guys gives me a "permission denied" error. I spent hours trying to figure this out, putting everything back the way I started - no go. I tried CheckTLS.com, and now it's showing "tls not available"? Up 'til now I've been using my 'sendmail-restart' which is in essence a 'kill -HUP'. I then tried a service sendmail stop then start, and now CheckTLS is working again! Fuck. So the moral of the story is to completely stop/start sendmail, don't just try to restart.

I'm still seeing a mountain of delayed connections trying to retry and still getting "permission denied". I'm guessing that these were sent while things were "broken" and will continue to retry until they fail, as they think they should go without STARTTLS as it wasn't available when they were sent.

Nope - that wasn't it. I should have thought harder about the fact that it was a "permission denied" error. I was assuming it was related to sendmail trying to access one of the files I'd messed with. Nope, it was the socket having the problem. openssl -connect commands were failing with this error as well. Problem was I got trapped in my own damn firewall. ipfw was blocking port25. I know I put the "always allow" for this servers IP address into the ipfw config file, now I gotta go figure out why that's not working. Fucking wasted day. Fix:
  ipfw table port25 del 207.229.65.53

Ok, so in the end, the fix I tried out but that got lost in the firewall mess up, was very simple. See STARTTLS howto for details, in a nutshell the /etc/mail/[systemname].mc file, the confCACERT_PATH should point to your /etc/ssl/certs dir.

1. Make new dir for this year.
   
   1. At NetSol download the "SSL package", not the latter the purports to be used for Apache.
   2. Unzip NetSol file.
   3. create a link to, or copy .key file from previous year into this dir. (Of course this assumes you did not make a new CSR and such, just renewing an existing cert. Otherwise, make sure the new signing key is in place here)


2. If you omit the following ov_chain step, you'll get unable to get local issuer certificate error.
   
   1. cp SSL_WILDCARD_CerticateAuthorityRoot.crt ov_chain.crt
      edit ov_chain.crt
   2. add a trailing newline
   3. append SSL_WILDCARD_IntermediateCA_2.crt
   4. add a trailing newline
   5. append SSL_WILDCARD_IntermediateCA_3.crt
   6. add a trailing newline


3. Edit /usr/local/etc/apache/Includes/httpd-ssl.conf
   
   Don't forget that there may be multiple blocks of these directives
   1. SSLCertificateFile set to full path to the new STAR.VINTNERS.NET.crt file
   2. SSLCACertificateChainFile set to full path to this ov_chain.crt
   3. SSLCACertificateFile set to full path to SSL_WILDCARD_CertificationAuthorityRoot.crt file


4. Restart apache:
   apachectl restart


5. Test Apache:
   
   1. In browser, visit said domain.
   2. Click on padlock icon.
   3. Click on "Connection Secure" item.
   4. Click on "More Information" item.
   5. This brings up a new window, click on "View Certificate" button.
   6. This brings up another window, check "Validity" section for new date.


6. Edit /etc/mail/[systemname].mc
   Change CERT_DIR to proper year, everything else should just work from this if you haven't changed filenames since last time.
   And of course make .cf file, copy into place and restart sendmail.


7. Test sendmail:
   
   1. send an email to that domain.
   2. check /var/log/maillog for errors.


8. Update imap/pop3/lmtp:
   Edit /usr/local/etc/imapd:
   
   1. Fix tls_server_cert - should match sendmail confSERVER_CERT
   2. Fix tls_server_key - should match sendmail confSERVER_KEY
   3. Fix tls_server_ca_file - should match sendmail confCACERT
   4. Fix tls_server_ca_dir - should match sendmail CERT_DIR
   Note! Thunderbird simply showed no new email even though I know there's email. In order to localize the problem, I installed Thunderbird on a different system - there it worked just fine. I messed with my TB settings, deleted certs, etc. no dice. Finally, I simply "removed" that "account", and recreated it - all is well now.