Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate = "${acme_certificate.ci.certificate_pem}"

  lifecycle {
    create_before_destroy = true

After some digging I’ve found this issue. Turns out that whatever lib fly is using to verify the certificates requires full chain of trust to be present. Surely enough running SSLabs on my domain showed:

Certificates provided      1 (1367 bytes)
Chain issues               Incomplete

Turns out acme_certificate.certificate_pem doesn’t produce a full chain certificate. Fortunately we can simply concatenate it like so:

resource "google_compute_ssl_certificate" "ci" {
  certificate = "${acme_certificate.ci.certificate_pem}${acme_certificate.ci.issuer_pem}"

After a brief terraform apply SSLabs showed:

Certificates provided   2 (2541 bytes)
Chain issues           None

And fly was once again working correctly. Hope this helps someone if they run into same issues.


Now read this

What’s making me happy 14-05-2017

Saw Green Day in concert this week, they really know how to put on a show, enjoyed it immensely. Took a drive out to Newcastle with Cassie, stopping by the 7-mile beach and Caves beach on our way. It was a nice day trip to get away from... Continue →