Temikus' ramblings

I.T. Engineer. Open Source contributor. Travel enthusiast.

Read this first

GCP SSH autocomplete with FZF and ZSH

I got inspired by AffectV post on AWS SSH completion and decided to write something similar for GCP.

Here’s the end result:
asciicast

Prerequisites

  • You use zsh as your main shell
  • You have fzf and its completion set up

Writing the module

First, we need to figure out how to get the list of information we want. gcloud provides a somewhat sensible formatting and filtering options for that, e.g. if I want to get the name, zone and external IP of the instance in a flat tab-separated table, we’ll have to set options like so:

λ gcloud compute instances list \
                         --format="table[no-heading](name,zone,status,networkInterfaces[0].accessConfigs[0].natIP)" \
                         --filter="status=RUNNING"
my-web-v37r              australia-southeast1-c  RUNNING  XX.XXX.XX.XX
my-web-wp2d              australia-southeast1-c  RUNNING  XX.XXX.XX.XX
my-worker-b96q
...

Continue reading →


ST3 - Remove Sublime Merge from Sidebar

Sublime Text 3 is my casual editor of choice.

They recently started pushing SublimeMerge which is a GUI Git Client. I’m a fan of Fork so I’m not really keen on the cluttering the context menu with things I won’t use.

I figured out how to remove them and hope the instructions may help others.

1) Install PackageResourceViewer package

2) Type a sublime prompt command (Cmd/Ctrl + P):

PackageResourceViewer: Open Resource

3) Choose Default -> Side Bar.sublime-menu

4) You can now edit the context menu and trim the unneeded options. For example:

    [
    { "caption": "New File", "command": "new_file_at", "args": {"dirs": []} },
    { "caption": "Rename…", "command": "rename_path", "args": {"paths": []} },
    { "caption": "Delete File", "command": "delete_file", "args": {"files": []} },
    { "caption": "Open Containing Folder…", "command": "open_containing_folder", "args": {"files":
...

Continue reading →


Tip: ssh `Match exec` and config for cloud instances

If you work a lot with ephemeral instances you will eventually run into this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Today I learned about Match exec config option in ~/.ssh/config which allows you to set certain options depending on custom command results.

Since all GCE external IP’s are routed through google LB’s, which resolve to *.bc.googleusercontent.com, we can use that to our advantage to disable recording GCE instances into known_hosts by putting the following into ~/.ssh/config:

 Do not record GCE instances into known_hosts file
Match exec "nslookup %h | grep 'bc.googleusercontent.com'"
   UserKnownHostsFile none

I hope others may find this useful too!

Continue reading →


Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate =
...

Continue reading →


Ansible  -  GCP dynamic inventory 2.0

The GCP module for Ansible has recently been updated to not rely on libcloud but the docs still continue to be slightly confusing, so I’m publishing a new article on how to set up Ansible GCP dynamic inventory.

As last time this assumes you already have:

  • Ansible installed (version >= 2.7)
  • A working python installation
  • Have a service account with Compute Engine API access
  • Exported a JSON key in Credentials subsection of your console

First, install “requests” and “google-auth” packages.

pip install requests google-auth

Note: if you use Homebrew, you may need to install the module via the built-in python installation that Ansible uses, e.g.:

λ ansible --version | grep "python.*location"

ansible python module location = /usr/local/Cellar/ansible/2.7.8/libexec/lib/python3.7/site-packages/ansible

λ /usr/local/Cellar/ansible/2.7.8/libexec/bin/pip install requests google-auth

Then...

Continue reading →


Reading large files in Ruby

I needed to slurp up some very large files into a ruby app recently and noticed some interesting behaviour in IO.foreach method.

While it is supposed to read file line by line without loading it up into memory, memory usage is quite significant compared to reading the files via an offset (IO.read).

Investigation

10 MB test file:

λ ls -alh xaa
-rw-r--r--@ 1 temikus  staff   9.5M Feb 18 11:24 xaa

Test script:

!/usr/bin/env ruby
require "benchmark/memory"

BUFFER = 4096

Benchmark.memory do |x|

  x.report("foreach") {
    File.foreach(ARGV[0]) do |line|
      line
    end
  }

  x.report("stream.read") {
    stream = File.new(ARGV[0])

    until stream.eof?
      stream.read(BUFFER)
    end
  }

  x.compare!
end

IO.foreach is using 20x memory:

λ ./cat_compare.rb xaa
Calculating -------------------------------------
             foreach   200.008M memsize (     0.000  retained)
...

Continue reading →


Ansible - Bootstrap python

When you’re starting hosts from scratch, you can avoid preinstalling python in your image or startup scripts by issuing a raw command in ansible in a pre_tasks section. This doesn’t require python to be available on the remote host, effectively bootstrapping it to continue with the playbook.

Here’s an example for debian-based hosts:

- hosts: my_host
  remote_user: my_user
  become: yes
  gather_facts: false

  pre_tasks:
  - name: Bootsrap Python2 for Ansible
    raw: bash -c "test -e /usr/bin/python || (apt -qqy update && apt install -qqy python-minimal)"
    register: output
    changed_when: output.stdout != ""
  - name: Gathering Facts
    setup:  Also known as gather_facts

Note: gather_facts is set to false since some advanced facts require running python code on the remote host, hence we disable it in the beginning and then re-run it using setup:

Continue reading →


Retrieving GCE metadata map

GCE metadata can be a useful tool for getting the information about the running instance.

One needs to know exactly where to look for a particular key however. For that, the metadata actually supports a ?recursive=true parameter, returning a json of all the data available.

Observe:

 Quick helper function to pretty print JSON 
$ alias to_j="ruby -e \"require 'json';puts JSON.pretty_generate(JSON.parse(STDIN.read))\""

$ curl -s http://metadata/computeMetadata/v1/instance/?recursive=true -H "Metadata-Flavor: Google" |  to_j
{
  "attributes": {
  },
  "cpuPlatform": "Intel Broadwell",
  "description": "",
  "disks": [
    {
      "deviceName": "persistent-disk-0",
      "index": 0,
      "mode": "READ_WRITE",
      "type": "PERSISTENT-SSD"
    }
  ],
  "hostname": "vagrant-test.c.graphite-sandbox.internal",
  "id": 77747111415725975,
  "image":
...

Continue reading →


Ansible - GCP Dynamic inventory bootstrap

Ansible GCP module bootstrap

The official docs on Ansible GCP module are quite a bit confusing, so I’ve decided to publish some steps to quickly bootstrap a working environment.

Main concept is that Ansible can not only use static files as a source of inventory but also execute a script and use it’s output as an in-memory representation of it. This allows us to make dynamic inventories to load up hosts from your cloud provider, for example. And, conveniently, there’s such a script in the Ansible’s contrib directory named gce.py.

This assumes you already have:

  • Ansible installed
  • A working python installation
  • Have a service account with Compute Engine API access
  • Exported a JSON key in Credentials subsection of your console

Here are quick bootstrap instructions on how to get it working:

Install libcloud and libcrypto:

λ pip install apache-libcloud pycrypto

Create a working dir:

...

Continue reading →


What’s making me happy 08-07-2017

It’s been a while since I wrote one of those - need to keep on track.

USA trip turned out to be a success - got a ton of work done, did quite a bit of progress on my projects, met with dear friends. Nevertheless, I’m very-very happy to return back to Sydney. Missed it (and more importantly Cassie) a whole lot.

Continuing trying to figure out my allergies. Waiting for the blood panel results, new medication seems to be working a bit better *knocks on wood*. Even though there’s no substantial progress in figuring out what it is, it’s still feels good to not be standing still.

Got quite a chuckle out of Louis Rossman’s review of the Surface Laptop construction: “You have to heat gun the laptop… If they included graphics chips in there - it’s now a two-for-one repair!”

Cap your ticket backlog is an interesting article on keeping the list of work items short and some ideas how to address...

Continue reading →