Mustang update #5

Battery tray, starter solenoid, radiator, and overflow tank all pulled out. Some of wiring loom has been partially extracted. Blower box also pulled. It’s heavier than I expected. Lots of metal in it. And, with the blower out, that means, I can get a first good look at the right hand cowl. Hmm, not looking great. Lots of nasty looking rust. Also, you can see something has poked a hole in the forward facing part of the footwell.

Mustang update #4

Removing the dashpad. I don’t know what these little end pieces of the piping for the interior air are made of but they basically disintegrated and turned to dust as soon as I touched them. Also the dashpad was glued down in a few spots which made it tricky to remove. Luckily the old one was already split so I wasn’t planning to try and keep it. All of the carpet is out now.

Mustang update #3

Left hand fender off. Looks like a nasty rust hole just under the windscreen. Also some bubbling of the paint in this area. I was getting some damp in the footwell on the driver side - this would explain that. Doors coming off. Console coming out. Seats coming out. Taking up the carpet in the back. --

Mustang update #2

Removing the valance, bumper and so on. Right hand fender removed. Love this hood latch. A look at the left and right hand side of the engine bay. The exhaust header is in pretty terrible shape. The brake booster is not much better.

Mustang update #1

A look at the engine bay at the very beginning. Lots of rust. I was having problems with the distributor/coil. So, I ended up dropping in a HEI. I had it on hand, but didn’t expect to use it until later. Hood and front grill come off. Boot lid off. Here’s a close up look at the left and right sides of the inside of the boot.

Using git hooks to trigger Rundeck jobs

At work, we keep our hiera yaml files in a git repo (encrypted using the excellent hiera-eyaml backend). I got really tired of doing a git pull on the puppetmaster each time I made a change to a hiera file. So, I wanted to set up a way to pull these changes automatically each time I did a commit. The first thought I had was to just set up some keys so the git server could ssh to the puppet master and run a git pull.

Growing the disk in a vagrant basebox

I love using vagrant when making big changes to my puppet code. It allows you to really easily start with a blank slate for each iteration. It gives me a reasonable amount of certainty that should^H^H^H when one my instances blow up, a single puppet pass will do everything needed to generate a new one. So, today, I’m happily working away on a major restructure of some puppet code when I run into this error.

Some thoughts on toil

The importance of distinguishing toil from other types of work and understanding its impact didn’t really click with me until recently when I read the Google SRE book. Toil is an insidious type of work. It’s impossible to completely eliminate, can be difficult to spot, and can have a huge impact on your productivity. If left unchecked, it can grow to consume the majority of your time meaning you’re left with little or no capacity for work which improves your environment.

Writing About Things Going Wrong

In this post, I want to talk about one very important part of dealing with an outage or some other type of undesirable event - doing a write-up afterwards. As well as being a good exercise in introspecting behaviours in your team, a well-written report can have a huge impact on how your work, as an ops/sysadmin/SRE/whatever person, is perceived outside of your team. Whether you think it is or not, giving other people in your organisation a view into some of the details of what you do is a part of your job.

Controlled Server Demolition

Today I tasked with one of those jobs which don’t come up very often - shutting down a service for good. This particular retirement posed an interesting challenge though. As per usual, I wanted to make sure that all traces of customer data was wiped from the systems, and, indeed, any of our own keys/users/passwords and so on. The challenge was, though, that I only had ssh access to the systems.