When you’re following an angel,
Does it mean you have to throw your body off a building?
Somewhere they’re meeting on a pinhead,
Calling you an angel,
Calling you the nicest things.
I heard they had a space program-
When they sing you can’t hear, there’s no air.
The day started off much like every other; if anything a little easier, in fact. D had taken the day off to take his kids to the seaside, instantly clearing my calendar of a whole bunch of the meetings I’d had planned for the day. Unfortunately that was the end of things going to plan. By 10am, I was having a catch up with one of my team when I got a rather urgent ping from W, saying “Hey, can you join this incident bridge? There’s a problem with the outbound proxies, bits of the front end aren’t working…” I dropped everything and joined, and sure enough, there was a problem somewhere; no traffic was being recorded. Working my way through the stack (which a since-departed member of my team implemented), I drilled into what had happened.
Huh. Someone deployed a change which broke a bunch of stuff, including deleting and recreating the Transit Gateway through which traffic arrives into that VPC. On top of that, there seemed to have been manual changes somewhere along the way to kludge the whole thing together. No wonder nothing worked, there were references to the old, now deleted, gateway, in various other AWS accounts. Spent a good couple of hours finding all the references to the now missing gateway and getting them updated, at which point everything sprang back into life and we could relax, but only a little. The rest of the afternoon was spent making recommendations on what to do to avoid a repeat; first and foremost cutting through the complete nonsense we’ve got from the person who made the change. Yes, there’s blame enough for everyone to go around, but claiming you didn’t make a production change when there’s a record of you pressing the button is a bit rich.
Took Sleeper, Jr. out for a ride in the evening, that helped a little. We’re doing about six miles in our hour out on the bike; it’s funny how much more terrifying 15 mph is when you’re accompanying an eight-year-old. Packed him off to bed and watched the final (so far) episode of Westworld. More stuff I wasn’t expecting, but… yeah, the next season is going to be interesting, especially given the post-credits scene.
Started the day with a meeting with D and the rest of the leadership, giving him the details he needs to defend what happened yesterday to his boss and explain in as non-political and face-saving ways as possible. The day didn’t get any better as I got requests coming in from a certain team to get them out of a hole; they have a requirement on my team to implement something that they haven’t raised with anyone and they urgently needed it. I wasn’t inclined to help, not least because the request was incredibly vague and sounded like something that could take a few days to implement.
Sure enough, not long later, I got a prod from someone more senior. Can I make this happen? Well, no, not in the time she’s talking about. I send the headache D’s way and give him a heads-up about it. By the time 4pm arrives, we’ve gone all the way we can with it and in a meeting I agree to do the absolute bare minimum they need to get it working, by the end of Friday, adequately specified. Everyone goes away, if not happy, then at least agreeing that progress has been made.
After a day like that I needed a clear head so took the bike out for a spin a little after 8pm, while J attended a yoga class. First mistake of the day was to forget my lights, significantly limiting the time I was willing to stay out. Second was to have new cleats fitted to my shoes; these ones haven’t seen thousands of miles of riding so aren’t completely loose like the old ones were. They needed the tension slackening in the pedals, which I’d completely forgotten to do. I managed to clip in once, and then after a mile of completely failing to repeat the action after stopping at lights, took a pit stop and slackened the springs, which are still harder to clip in than I’d expected. Got home ten miles down, 40 minutes later. J finished her class and headed up to bed, feeling a little dizzy and light headed. I followed a little later.
Up at the usual time; J seemed to be feeling a little better but still dizzy when I headed into my first (2 hour) meeting of the day; by the time I got out, she was feeling much worse. I cut short my work day, just finishing the bits and pieces I’d agreed the day before, and getting into an argument with the same team as yesterday that no, I they couldn’t have yet another new thing they’d asked for, especially one of such complexity. Left strict instructions not to even engage with that request to my team member who was being asked for advice, and got the hell out of there.
Had a lovely afternoon with the kids, mostly playing board games with A, while Sleeper Jr. tidied his room. We went out in the garden and tinkered with bikes, then I took both the kids for a walk before heading home and cooking dinner – curry. I’d planned on enchiladas but hadn’t reckoned on the grated cheese I’d been planning on using being mouldy. Next time….
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.