Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate = "${acme_certificate.ci.certificate_pem}"

  lifecycle {
    create_before_destroy = true

After some digging I’ve found this issue. Turns out that whatever lib fly is using to verify the certificates requires full chain of trust to be present. Surely enough running SSLabs on my domain showed:

Certificates provided      1 (1367 bytes)
Chain issues               Incomplete

Turns out acme_certificate.certificate_pem doesn’t produce a full chain certificate. Fortunately we can simply concatenate it like so:

resource "google_compute_ssl_certificate" "ci" {
  certificate = "${acme_certificate.ci.certificate_pem}${acme_certificate.ci.issuer_pem}"

After a brief terraform apply SSLabs showed:

Certificates provided   2 (2541 bytes)
Chain issues           None

And fly was once again working correctly. Hope this helps someone if they run into same issues.


Now read this

Reading large files in Ruby

I needed to slurp up some very large files into a ruby app recently and noticed some interesting behaviour in IO.foreach method. While it is supposed to read file line by line without loading it up into memory, memory usage is quite... Continue →