Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate = "${acme_certificate.ci.certificate_pem}"

  lifecycle {
    create_before_destroy = true

After some digging I’ve found this issue. Turns out that whatever lib fly is using to verify the certificates requires full chain of trust to be present. Surely enough running SSLabs on my domain showed:

Certificates provided      1 (1367 bytes)
Chain issues               Incomplete

Turns out acme_certificate.certificate_pem doesn’t produce a full chain certificate. Fortunately we can simply concatenate it like so:

resource "google_compute_ssl_certificate" "ci" {
  certificate = "${acme_certificate.ci.certificate_pem}${acme_certificate.ci.issuer_pem}"

After a brief terraform apply SSLabs showed:

Certificates provided   2 (2541 bytes)
Chain issues           None

And fly was once again working correctly. Hope this helps someone if they run into same issues.


Now read this

GCP SSH autocomplete with FZF and ZSH

I got inspired by AffectV post on AWS SSH completion and decided to write something similar for GCP. Here’s the end result: Prerequisites # You use zsh as your main shell You have fzf and its completion set up Writing the module # First,... Continue →