Fixing the strange chef.io SSL errors when using Chef Provisioning08 May 2015
TLDR: Install the ChefDK for great good.
Today I was scheduled to work with another developer on Chef Provisioning on AWS so I decided to dust off a few Chef Provisioning recipes I had lying around from a demo a few months back. When I tried to provision one of them I was greeted with a rather alarming problem
ERROR: SSL Validation failure connecting to host: www.chef.io - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Interesting. Why are we contacting chef.io and why can’t I validate the SSL
certificate? After running
chef-client with debug logging I determined that
Chef Provisioning was attempting to download and cache the chef-client omnibus
package for the machines that I was provisioning. The second issue is a much
bigger problem, but before we go any further I have to mention that this wouldn’t
have been a problem at all if my
chef-client had been installed using the
As as Ruby developer, I find myself using chruby to bounce between versions and
so I was using Chef the old-school manual gem install way. Big mistake.
Rather than switch to the ChefDK and get back to productive work, I decided to hunt down the root cause. After verifying that the latest OpenSSL version was installed and linked, my system gems were updated, and that I’d installed the latest security bundle from Apple, I figured it must be a ca-cacert issue.
First I determined which cert Ruby was using
$: ruby -r openssl -e "p OpenSSL::X509::DEFAULT_CERT_FILE" "/usr/local/etc/openssl/cert.pem"
Then updated updated it to reflect Apple’s latest certs
$: security find-certificate -a -p /Library/Keychains/System.keychain > /usr/local/etc/openssl/cert.pem $: security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain >> /usr/local/etc/openssl/cert.pem
Then tried running the provisioner again
$: chef-client -z provisioner.rb ... ERROR: SSL Validation failure connecting to host: www.chef.io - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Damn. Okay, maybe Apple’s certs are no good. Let’s try the latest bundle from the curl/Mozilla folks.
$: curl http://curl.haxx.se/ca/cacert.pem -o /usr/local/etc/openssl/cert.pem $: chef-client -z provisioner.rb ... ERROR: SSL Validation failure connecting to host: opscode-omnibus-packages.s3.amazonaws.com - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificat e verify failed
Sweet, certificate validation worked on chef.io but now we’re unable to verify the S3 cert. We’re stuck in limbo here because Apple’s bundle doesn’t like chef.io/fastly’s cert and curl’s bundle doesn’t like S3. After trying to verify the S3 cert I stumbled upon a forum post by Lamont that details that support for 1024 bit signing keys has been phased out. Yikes. Until AWS reissues their cert with a proper root signature we either have to abandon S3 or use an older certificate bundle, the latter of which is how we’ve solved that problem in the ChefDK.
I snagged a copy of the certificate bundle from last August (before 1024 bit keys were phased out) and gave it a go
$: curl https://s3.amazonaws.com/uploads.hipchat.com/7557/2027616/rPfbosc2b4pPNh8/cacert.pem -o /usr/local/etc/openssl/cert.pem $: chef-client -z provisioner.rb ... - create new file /Users/ryan/.chef/package_cache/chef_12.3.0-1_amd64.deb ...
Viola! Using the old bundle obviously isn’t ideal, but it unblocked me this time.