Temikus' ramblings

I.T. Engineer. Open Source contributor. Travel enthusiast.

Read this first

Tip: ssh `Match exec` and config for cloud instances

If you work a lot with ephemeral instances you will eventually run into this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Today I learned about Match exec config option in ~/.ssh/config which allows you to set certain options depending on custom command results.

Since all GCE external IP’s are routed through google LB’s, which resolve to *.bc.googleusercontent.com, we can use that to our advantage to disable recording GCE instances into known_hosts by putting the following into ~/.ssh/config:

 Do not record GCE instances into known_hosts file
Match exec "nslookup %h | grep 'bc.googleusercontent.com'"
   UserKnownHostsFile none

I hope others may find this useful too!

Continue reading →


Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate =
...

Continue reading →


Ansible  -  GCP dynamic inventory 2.0

The GCP module for Ansible has recently been updated to not rely on libcloud but the docs still continue to be slightly confusing, so I’m publishing a new article on how to set up Ansible GCP dynamic inventory.

As last time this assumes you already have:

  • Ansible installed (version >= 2.7)
  • A working python installation
  • Have a service account with Compute Engine API access
  • Exported a JSON key in Credentials subsection of your console

First, install “requests” and “google-auth” packages.

pip install requests google-auth

Note: if you use Homebrew, you may need to install the module via the built-in python installation that Ansible uses, e.g.:

λ ansible --version | grep "python.*location"

ansible python module location = /usr/local/Cellar/ansible/2.7.8/libexec/lib/python3.7/site-packages/ansible

λ /usr/local/Cellar/ansible/2.7.8/libexec/bin/pip install requests google-auth

Then...

Continue reading →


Reading large files in Ruby

I needed to slurp up some very large files into a ruby app recently and noticed some interesting behaviour in IO.foreach method.

While it is supposed to read file line by line without loading it up into memory, memory usage is quite significant compared to reading the files via an offset (IO.read).

Investigation

10 MB test file:

λ ls -alh xaa
-rw-r--r--@ 1 temikus  staff   9.5M Feb 18 11:24 xaa

Test script:

!/usr/bin/env ruby
require "benchmark/memory"

BUFFER = 4096

Benchmark.memory do |x|

  x.report("foreach") {
    File.foreach(ARGV[0]) do |line|
      line
    end
  }

  x.report("stream.read") {
    stream = File.new(ARGV[0])

    until stream.eof?
      stream.read(BUFFER)
    end
  }

  x.compare!
end

IO.foreach is using 20x memory:

λ ./cat_compare.rb xaa
Calculating -------------------------------------
             foreach   200.008M memsize (     0.000  retained)
...

Continue reading →


Ansible - Bootstrap python

When you’re starting hosts from scratch, you can avoid preinstalling python in your image or startup scripts by issuing a raw command in ansible in a pre_tasks section. This doesn’t require python to be available on the remote host, effectively bootstrapping it to continue with the playbook.

Here’s an example for debian-based hosts:

- hosts: my_host
  remote_user: my_user
  become: yes
  gather_facts: false

  pre_tasks:
  - name: Bootsrap Python2 for Ansible
    raw: bash -c "test -e /usr/bin/python || (apt -qqy update && apt install -qqy python-minimal)"
    register: output
    changed_when: output.stdout != ""
  - name: Gathering Facts
    setup:  Also known as gather_facts

Note: gather_facts is set to false since some advanced facts require running python code on the remote host, hence we disable it in the beginning and then re-run it using setup:

Continue reading →


Retrieving GCE metadata map

GCE metadata can be a useful tool for getting the information about the running instance.

One needs to know exactly where to look for a particular key however. For that, the metadata actually supports a ?recursive=true parameter, returning a json of all the data available.

Observe:

 Quick helper function to pretty print JSON 
$ alias to_j="ruby -e \"require 'json';puts JSON.pretty_generate(JSON.parse(STDIN.read))\""

$ curl -s http://metadata/computeMetadata/v1/instance/?recursive=true -H "Metadata-Flavor: Google" |  to_j
{
  "attributes": {
  },
  "cpuPlatform": "Intel Broadwell",
  "description": "",
  "disks": [
    {
      "deviceName": "persistent-disk-0",
      "index": 0,
      "mode": "READ_WRITE",
      "type": "PERSISTENT-SSD"
    }
  ],
  "hostname": "vagrant-test.c.graphite-sandbox.internal",
  "id": 77747111415725975,
  "image":
...

Continue reading →


Ansible - GCP Dynamic inventory bootstrap

Ansible GCP module bootstrap

The official docs on Ansible GCP module are quite a bit confusing, so I’ve decided to publish some steps to quickly bootstrap a working environment.

Main concept is that Ansible can not only use static files as a source of inventory but also execute a script and use it’s output as an in-memory representation of it. This allows us to make dynamic inventories to load up hosts from your cloud provider, for example. And, conveniently, there’s such a script in the Ansible’s contrib directory named gce.py.

This assumes you already have:

  • Ansible installed
  • A working python installation
  • Have a service account with Compute Engine API access
  • Exported a JSON key in Credentials subsection of your console

Here are quick bootstrap instructions on how to get it working:

Install libcloud and libcrypto:

λ pip install apache-libcloud pycrypto

Create a working dir:

...

Continue reading →


What’s making me happy 08-07-2017

It’s been a while since I wrote one of those - need to keep on track.

USA trip turned out to be a success - got a ton of work done, did quite a bit of progress on my projects, met with dear friends. Nevertheless, I’m very-very happy to return back to Sydney. Missed it (and more importantly Cassie) a whole lot.

Continuing trying to figure out my allergies. Waiting for the blood panel results, new medication seems to be working a bit better *knocks on wood*. Even though there’s no substantial progress in figuring out what it is, it’s still feels good to not be standing still.

Got quite a chuckle out of Louis Rossman’s review of the Surface Laptop construction: “You have to heat gun the laptop… If they included graphics chips in there - it’s now a two-for-one repair!”

Cap your ticket backlog is an interesting article on keeping the list of work items short and some ideas how to address...

Continue reading →


What’s making me happy 18-06-2017

Visiting NorCal and Seattle area over the past 2 weeks, it’s been nice to visit some of my old friends. Those moments, although brief do warm my heart.

Took some time walking around Stanford. The Dish trail turned out to be a great walk.
19146149_10209709636086184_430125816262029059_n.jpg

Watched Hamilton in SF with my good friend Ben - it is indeed a great and very modern musical. Can highly recommend it.

Museum of Pop Culture in Seattle turned out to be a very unexpected gem. It’s a great collection of Music, Sci-Fi and other memorabilia. Their Guitar exhibition alone is worth it.

Visited Fremont Fair over the weekend - a very quirky Art fair with a nice vibe to it. Didn’t buy anything but it was nice to stroll around and take in the atmosphere.

New installment of Fire Emblem series is out with Fire Emblem: Echoes - already got a copy - can’t wait to play it!

Humble Bundle - Sekai Project is out. A bunch of great VN’s for a good...

Continue reading →


What’s making me happy 05-06-2017

Post is almost on time if you count in PST :D

Started running again for the first time in 5 months - it was only 2k and my time wasn’t the best, but it feels so good to get back in the game.

Went to Jenolan caves over the previous weekend - amazing cave system with a lot of stuff to explore in the Blue Mountains
jenolan-karst-12.jpg

Cassie introduced me to Spawn Point, which turned out to be a cosy geeky bar in central Sydney. Much alcohol-fueled Mario Kart ensued.

Holy cow, it actually happened - Utawarerumono: Mask of Deception is out in English. It’s a strange combination of a Visual Novel and a tactical RPG but the story is excellent. Playing/Watching the first one is highly recommended before starting this one.

Discovered Nidhogg - a fast-paced multiplayer fighting arcade with beautiful retro visuals and fast responsive mechanics.

Falsehoods programmers believe about time is an oldie, but I still...

Continue reading →