Artem's ramblings

My blog about Engineering, Management, Reliability and whatever my magpie brain is focusing at the moment.

Read this first

Manager hour - comms during turbulent times

Generally, people have a finite capacity to “deal with things”. Organisational events and changes exhaust that finite capacity, which makes people over-evaluate any further changes to their work environment as they crave stability.

As such, after significant changes such as layoffs, reorgs, pivots, etc., it’s even more critical to over-communicate and communicate with care when doing announcements.

And the most important thing here is follow-up! No one does it 100% right all the time. It’s whether you recognise something is missing and follow up that matters much more.

View →


Management Hour - Interviewing

Interviews should be:

  • Consistent
  • Fair
  • Humane

As such, here are some things to keep in mind:

  • Ensure that at least 80% of your questions overlap between different candidates. Otherwise, you create an uneven playing field.
  • A memory of an experience fades extremely quickly, so make sure to jot down your feedback ASAP—ideally, on the same day. I usually block off my calendar right after the interview for feedback.
  • If you interview in tandem - discuss the candidate after you submit the feedback so you don’t bias each other. If your scores are radically different - it will at least create a learning opportunity for both.
  • If you know that you will be biased for or against the candidate - make sure you interview at least in tandem or recuse yourself from the process.
  • No trick or “Aha!” questions should be present in interviews. They just bin candidates into 2 buckets: those who know and do...

Continue reading →


SRE Consulting engagements

Due to extreme demand on my Office Hours I’m opening a paid consulting stream for those that want more of my time and are willing to pay for it - i.e. SRE leaders, Seasoned Professionals and Companies that need advice.

  • Are you a seasoned professional new to the SRE role?
  • Perhaps you lead a team that struggles with a particular operational or culture problem?
  • Are you a startup that needs a nudge in the right direction to scale?
  • Do you need in-depth assistance with one of the OSS projects I maintain?

Then this is for you: https://calendly.com/temikus/consulting
10% of all proceeds go to charity ❤️ https://www.fame.org.au/
Questions? consulting < at > temik.me

FAQ

Q: Are you stopping your free office hours?
A: No, I’m still scheduling those in the same manner for the foreseeable future as my reasoning hasn’t changed.

Q: Why are you doing this?
A: Popular demand. I’m often getting...

Continue reading →


Office Hours

Following the example of my friend natwelch@ I decided to try my hand at public office hours.

The idea is simple - I have a 60 minute weekly slot that anyone can book through calendly and we can do anything of the following via videochat:

  • Helping people with non-IT/CS backgrounds get into the industry.

    Because we all stand on the shoulders of giants and I’m not an exception. I came to IT with no background and am largely self-taught, but I couldn’t do it without a lot of help from some wonderful folks ( Hi Nat! 👋).

  • Assisting OSS projects with CI/CD, Monitoring and Infrastructure.

    Because I ❤️ OpenSource and maintain a couple of projects w/associated infra myself.

  • Talking about anything that you think is “Ops” / “DevOps” or “Site Reliability Engineering”.

Since I work for a large company, there’s a small disclaimer:

  • All opinions expressed during the chats are my own and are...

Continue reading →


GCP SSH autocomplete with FZF and ZSH

I got inspired by AffectV post on AWS SSH completion and decided to write something similar for GCP.

Here’s the end result:
asciicast

Prerequisites

  • You use zsh as your main shell
  • You have fzf and its completion set up

Writing the module

First, we need to figure out how to get the list of information we want. gcloud provides a somewhat sensible formatting and filtering options for that, e.g. if I want to get the name, zone and external IP of the instance in a flat tab-separated table, we’ll have to set options like so:

λ gcloud compute instances list \
                         --format="table[no-heading](name,zone,status,networkInterfaces[0].accessConfigs[0].natIP)" \
                         --filter="status=RUNNING"
my-web-v37r              australia-southeast1-c  RUNNING  XX.XXX.XX.XX
my-web-wp2d              australia-southeast1-c  RUNNING  XX.XXX.XX.XX
my-worker-b96q
...

Continue reading →


ST3 - Remove Sublime Merge from Sidebar

Sublime Text 3 is my casual editor of choice.

They recently started pushing SublimeMerge which is a GUI Git Client. I’m a fan of Fork so I’m not really keen on the cluttering the context menu with things I won’t use.

I figured out how to remove them and hope the instructions may help others.

1) Install PackageResourceViewer package

2) Type a sublime prompt command (Cmd/Ctrl + P):

PackageResourceViewer: Open Resource

3) Choose Default -> Side Bar.sublime-menu

4) You can now edit the context menu and trim the unneeded options. For example:

    [
    { "caption": "New File", "command": "new_file_at", "args": {"dirs": []} },
    { "caption": "Rename…", "command": "rename_path", "args": {"paths": []} },
    { "caption": "Delete File", "command": "delete_file", "args": {"files": []} },
    { "caption": "Open Containing Folder…", "command": "open_containing_folder", "args": {"files":
...

Continue reading →


Tip: ssh `Match exec` and config for cloud instances

If you work a lot with ephemeral instances you will eventually run into this message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Today I learned about Match exec config option in ~/.ssh/config which allows you to set certain options depending on custom command results.

Since all GCE external IP’s are routed through google LB’s, which resolve to *.bc.googleusercontent.com, we can use that to our advantage to disable recording GCE instances into known_hosts by putting the following into ~/.ssh/config:

 Do not record GCE instances into known_hosts file
Match exec "nslookup %h | grep 'bc.googleusercontent.com'"
   UserKnownHostsFile none

I hope others may find this useful too!

Continue reading →


Terraform ACME provider full chain certificates

After rearchitecting my Concourse setup a bit to shard web workers I’ve noticed that fly stopped connecting:

λ fly login -t fog --concourse-url https://ci.saragosa.tv:443           (1)
logging in to team 'main'

could not reach the Concourse server called fog:

    Get https://ci.saragosa.tv:443/api/v1/info: x509: certificate signed by unknown authority

is the targeted Concourse running? better go catch it lol

Which was surprising to me since WebUI was working without any issues and I was getting green checks on the security tab of Chrome’s “Developer tools”

The important bit here is that I was using Terraform to generate acme certificates and pass them into GCP’s HTTPS LB:

resource "google_compute_ssl_certificate" "ci" {
  name = "my-ci-project-${random_id.generation.hex}"
  private_key = "${acme_certificate.ci.private_key_pem}"
  certificate =
...

Continue reading →


Ansible  -  GCP dynamic inventory 2.0

The GCP module for Ansible has recently been updated to not rely on libcloud but the docs still continue to be slightly confusing, so I’m publishing a new article on how to set up Ansible GCP dynamic inventory.

As last time this assumes you already have:

  • Ansible installed (version >= 2.7)
  • A working python installation
  • Have a service account with Compute Engine API access
  • Exported a JSON key in Credentials subsection of your console

First, install “requests” and “google-auth” packages.

pip install requests google-auth

Note: if you use Homebrew, you may need to install the module via the built-in python installation that Ansible uses, e.g.:

λ ansible --version | grep "python.*location"

ansible python module location = /usr/local/Cellar/ansible/2.7.8/libexec/lib/python3.7/site-packages/ansible

λ /usr/local/Cellar/ansible/2.7.8/libexec/bin/pip install requests google-auth

Then...

Continue reading →


Reading large files in Ruby

I needed to slurp up some very large files into a ruby app recently and noticed some interesting behaviour in IO.foreach method.

While it is supposed to read file line by line without loading it up into memory, memory usage is quite significant compared to reading the files via an offset (IO.read).

Investigation

10 MB test file:

λ ls -alh xaa
-rw-r--r--@ 1 temikus  staff   9.5M Feb 18 11:24 xaa

Test script:

!/usr/bin/env ruby
require "benchmark/memory"

BUFFER = 4096

Benchmark.memory do |x|

  x.report("foreach") {
    File.foreach(ARGV[0]) do |line|
      line
    end
  }

  x.report("stream.read") {
    stream = File.new(ARGV[0])

    until stream.eof?
      stream.read(BUFFER)
    end
  }

  x.compare!
end

IO.foreach is using 20x memory:

λ ./cat_compare.rb xaa
Calculating -------------------------------------
             foreach   200.008M memsize (     0.000  retained)
...

Continue reading →