Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate = "${acme_certificate.ci.certificate_pem}"

  lifecycle {
    create_before_destroy = true

After some digging I’ve found this issue. Turns out that whatever lib fly is using to verify the certificates requires full chain of trust to be present. Surely enough running SSLabs on my domain showed:

Certificates provided      1 (1367 bytes)
Chain issues               Incomplete

Turns out acme_certificate.certificate_pem doesn’t produce a full chain certificate. Fortunately we can simply concatenate it like so:

resource "google_compute_ssl_certificate" "ci" {
  certificate = "${acme_certificate.ci.certificate_pem}${acme_certificate.ci.issuer_pem}"

After a brief terraform apply SSLabs showed:

Certificates provided   2 (2541 bytes)
Chain issues           None

And fly was once again working correctly. Hope this helps someone if they run into same issues.


Now read this

What’s making me happy 18-06-2017

Visiting NorCal and Seattle area over the past 2 weeks, it’s been nice to visit some of my old friends. Those moments, although brief do warm my heart. Took some time walking around Stanford. The Dish trail turned out to be a great walk.... Continue →