A tablet by the door where members check themselves in. It started as a Next.js app, then became an iOS and an Android app that render exactly that page in a WebView and carry no product logic at all. The most instructive part was the subsystem I built, tested, shipped and deleted eight days later.

The check-in app is one web page and two wrappers

Codex has been part of my workflow for the last year. After a week with Astra, a tool I already used regularly feels exciting again. This model is a game changer for me.

A year with Codex, a week with Astra

We opened 1club to other people’s assistants through an MCP server: OAuth in front, a scope per capability, and every tool a thin door onto the platform API. What came out the other side was faster onboarding, reports our customers make themselves, and billing mistakes that get found before the customer does.

Onboarding took two days. Now it takes two hours.

A cloud platform needs to open a door on a private network it will never see. Why the on-premise half is Go, the protocol surprises that only turned up against real hardware, and the nine steps between a release artefact and a working site.

A door on someone else's network

In June I cut a Cloud Run bill from $58 to $13 by throttling idle CPU. This week I worked out what that boolean had actually been buying: every scheduled job in the environment, none of which had run since.

$58 a month to keep one container awake

Cloud Run without a startup probe checks that something bound the port. Three attempts later the API has a deep readiness gate, a shallow liveness check that does no I/O, and a draining answer — and the reason they must be different endpoints is that one of them restarts the container.

Ready is not alive

An org policy makes service-account JSON keys unmintable, so GitHub Actions federates instead. What Workload Identity Federation actually buys, the three identities a deploy needs, and the permission whose absence reads as "image not found".

Nobody has a service-account key

A new club arrives with a spreadsheet exported from whatever they used before, and the columns never match ours. We put a model in front of the import — but only in front of it. It reads five rows, proposes a mapping, and the thing that writes to the database has never heard of it.

The CSV importer that proposes and never executes

Our vectors live in the same PostgreSQL as the application, which is the right call until three clients want the same twenty connections. Last weekend the retrieval pool alone was sized to the whole role limit, and the vector store was quietly opening more. Notes on budgeting connections instead of adding a database.

The RAG client was stealing my connections

I designed a seven-project Google Cloud topology, then found out the billing account tops out at five. A month of support tickets did not move it. What the collapse to four projects actually cost, and what it did not.

The project quota that killed Shared VPC

Every staff-facing assistant in our admin product runs on the same forty lines: bind the tools, call the model, run what it asked for, feed the results back, stop at a cap. This week that loop grew a guard against retrying a call that already failed, and it removed the most embarrassing thing our assistants did.

Five assistants, one loop

NestJS 11 moved to Express 5 in January and Node 24 went LTS in October. What that upgrade actually cost, why I left the Fastify adapter alone, and why node src/main.ts will never run a Nest app.

Express 5, Fastify 5, and the adapter I did not swap

This year every coding assistant grew a file it reads before it starts: CLAUDE.md, GEMINI.md, .cursor/rules, and since August a shared AGENTS.md. Notes on what went into mine, what came back out, and why instructions for something with no memory are not documentation.

AGENTS.md, or writing for a reader that forgets

Microsoft is porting the TypeScript compiler to Go and reporting roughly ten times faster type-checking. The choice of language caused more discussion than the speedup, and the reasoning behind it is the genuinely interesting part.

They ported tsc to Go

Redis 8 added AGPLv3 alongside the two source-available licences, fourteen months after leaving BSD. That is the fourth licence event I have written about in two years, and the useful output is a checklist rather than an opinion.

Redis is open source again, and I have stopped keeping score

Bought packs, then Google Images, then a licensing position I could not defend. In June I replaced every block image on the site with generated art.

The block images are generated now

Six months after Anthropic published the Model Context Protocol, I spent a weekend writing a tiny server for this site. The protocol took an hour. Deciding which operations to expose took the rest of the weekend, and it is the only part worth writing about.

A tool call is just a schema

Kafka 4.0 shipped in March as the first release that cannot talk to ZooKeeper at all. Three posts and two and a half years after I started following this, the deprecation actually completed — which is rarer than it should be.

ZooKeeper is gone

The client library on top of the SNS fan-out. Why publish returns nothing, the two sets of promises a graceful shutdown has to drain, payloads that travel as pointers, and a consumer that puts its own failures in the dead-letter queue early.

A batch of ten, a flush timer, and the event that did not fit

Replacing a Redis job queue with SNS fan-out to SQS. Why every topic is FIFO, why the subscription has to be created in the topic region, and why the deny statement was the part we rewrote five times.

One topic, three regions, and the policy that took the longest

AWS previewed a distributed SQL database at re:Invent with active-active multi-region writes and strong consistency. The mechanism underneath is a hardware clock in every instance, which is a more interesting answer than the feature list.

Aurora DSQL, and the clock doing the hard part

Our MemoryDB clusters run Redis 7, and on AWS that line has nowhere left to go — 7.2 and above are Valkey only. A week of pricing the alternatives against a socket fan-out and a job queue, and the answer was to stay.

A week of alternatives, and the migration I did not make

Express leaves the interesting decisions to you, and every repository answers them differently. Two years in, the container is the part of NestJS I would keep — and Node 22 just went LTS.

NestJS over plain Express

The warehouse needed the operational database, and the operational database has no route in. Notes on a reverse SSH tunnel to a managed ingest vendor, why the target is an SRV record and only the first one, and the VPN box that ended up doing two jobs.

The tunnel that starts on our side

I listed three things PostgreSQL 15 MERGE could not do. PostgreSQL 17 does all three. What is more interesting is the one complaint it did not fix, because that one was never a missing feature.

Two years later, MERGE grew the clauses I wanted

Elastic added AGPL alongside SSPL and ELv2 in August, which makes Elasticsearch open source again by the usual definition. After a year of watching licences move the other way, this one went the other direction.

Elasticsearch came back

Embedding a BI tool in a multi-tenant product gives you two tenancy mechanisms that look identical in a demo. One is a URL parameter. The other is a SQL wildcard on a user attribute, and it is the only one holding.

The filter you can see is not the one that matters

Redis left BSD in March and Valkey existed within weeks, under the Linux Foundation with the cloud providers behind it. The second time you watch this happen, the interesting part is how much faster it went.

Redis changed its licence, and the fork was already warm

Proton renders Terraform into a PR and waits for a human. We replaced the human with validate, plan and Terratest, then merged automatically and applied on merge. What that requires, and what it costs.

Auto-merging a pull request nobody wrote

AWS Proton with Terraform means self-managed provisioning: it renders your template into a repository and submits a PR. What the template bundle looks like, what lands in the branch, and the callback you cannot forget.

Proton opens the pull request. You own everything after it.

Model names are global inside a BI instance, so two environments cannot both own the name. What promotion looks like when it is a rename, why the regex is anchored on known model names, and the naming rule that is the real product of the pipeline.

Promoting a dashboard is a string replacement

Redis left BSD on the twentieth of March. Eight days later the Linux Foundation announced a fork, carrying on from 7.2.4 with AWS, Google Cloud and Oracle behind it. Notes written while it is still a name and a repository.

Valkey exists now

A service catalog that spawns microservices: one form, one workflow_dispatch, and two very different things behind it. What belongs in orchestration, what needs infrastructure, and why the form is the real contract.

The button in Port and the service it makes

Go 1.22 made every iteration of a for loop create new variables, fixing a decade-old gotcha. Changing the meaning of existing code is the thing language teams are not supposed to do, so how they did it is more interesting than what they did.

Go changed a loop, and nothing broke

OpenTofu 1.6 reached general availability on the tenth of January, four months after the project had a name and no release. The interesting question was never whether the binary would work.

The fork shipped

The AOT numbers in the announcement are large enough to be suspicious, so I went back to the experiment I set up two years ago and ran it on a release that can finally produce a native executable.

.NET 8, Native AOT, and what a container actually costs

A GitHub Actions workflow that builds a throwaway Atlas cluster from last night's backup, strips the customers out of it with a Node script, and only then lets an engineer connect. The order of those steps is the whole design.

The snapshot restore was the easy half

The service was already on MongoDB, so Agenda would have cost no new infrastructure. A blocking pop, retries I did not have to write, and a five second poll interval decided otherwise.

BullMQ, and the Mongo queue I did not use

Moving a self-managed MongoDB replica set onto Atlas without a maintenance window. The private network went in first and Terraform owned it; mongomirror did the rest from the one box that could see both sides.

The cutover was a connection string

HashiCorp moved Terraform to the BUSL in August, and three days ago the fork became OpenTofu under the Linux Foundation. What the licence actually restricts, the sentence most of the panic skipped, and what I concluded for a team that only ever used Terraform to manage its own infrastructure.

The licence changed under the state file

Kafka 3.5 deprecated ZooKeeper and shipped the migration as early access. Eight months ago the answer for an existing cluster was "wait" — this is what waiting turned into, and it is a dual-write mode with a way back.

The bridge release arrived

Terraform 1.5 made import part of the plan, and added config generation. Every account I have worked in has a drawer of things created by hand in 2019, and until this month adopting them was too tedious to schedule.

The import block, and the resources nobody wrote down

Next.js 13.4 marked the App Router stable on the fourth of May. I spent a Saturday on a migration branch and then deleted it — not because the new model is wrong, but because the thing it is good at is not the thing this site does.

The App Router went stable. My site did not move.

I finally spent a weekend writing Rust rather than reading about it. Notes on what the borrow checker is actually for, why the compiler errors deserve their reputation, and the Cargo change that made the first five minutes bearable.

A weekend with Rust, and the index that stopped cloning

The decorators in your NestJS codebase are not the decorators TypeScript 5.0 shipped. Knowing which one you are using is the difference between a flag you can delete and a flag you must keep.

TypeScript 5.0 and the decorators that are finally the real ones

Go 1.20 shipped on the first of February with errors.Join and multiple %w verbs. A small change that finally made the error model click for someone arriving from C# and TypeScript — plus a preview of profile-guided optimisation I would measure before believing.

Errors are values. Go 1.20 made them a list.

Pipes went generally available at re:Invent with four stages and no code. Most of the Lambdas I have written between a queue and a target are exactly those four stages, badly.

EventBridge Pipes, and the Lambda I deleted

Kafka 3.3 marked KRaft production ready, which ends a long plan to delete a dependency. A weekend with a three-node quorum on a laptop, and a careful reading of what "production ready" was not claiming yet.

Kafka without ZooKeeper, for new clusters only

C# 11 landed with raw string literals, required members and list patterns. Three small features, and two of them remove a class of bug rather than a class of keystrokes.

.NET 7, and the string literal I stopped escaping

PostgreSQL 15 landed with the SQL-standard MERGE, and everyone read it as "Postgres finally has upsert". It is a different tool with a different failure mode, and the interesting difference is what happens when two of them run at once.

MERGE is not the upsert you already had

Auto-scaling GitHub Actions runners on EC2 spot, deployed once per product so a slow test suite in one place cannot starve a deploy somewhere else. What it cost us, and what I still had to own.

Self-hosted runners, one deployment per product

Change streams have always told you a document changed and what it became. 6.0 can also tell you what it was, which quietly removes a category of code I have written more than once.

MongoDB 6.0, and the change stream that can show you the before

Two dependencies became built-ins in April, both marked experimental. One of them I removed from a project the same week; the other I am leaving exactly where it is, and the reason is not the feature.

Node 18 shipped fetch and a test runner. I kept one.

Terraform 1.2 added precondition and postcondition blocks. They do not do anything you could not do with a clever variable validation and a lot of swearing — they just do it where the assumption actually lives.

Terraform learned to say what it assumed

Kubernetes 1.24 removed dockershim from the kubelet, and a lot of people read that as "Docker images stop working". They do not. What actually changed, and why I read the whole thing without running a cluster.

The dockershim went away and my images did not notice

Chemical names break naive search in about four ways. What Lucene gave us for free, which parts we actually had to write, and where it is the wrong answer.

Every string search I did not have to write

An NLP annotator in C#, a proposal to consolidate it onto the JVM, and the two numbers that ended the discussion. Mostly a note about where C# speed actually comes from.

The Java port we could not make fast enough

A YAML file, a managed runner, and ordinary Bash or JavaScript when the job gets interesting. Why GitHub Actions is my default for a repository already on GitHub.

GitHub Actions over Jenkins

Groovy 4.0 shipped last month. What the language actually is, how it runs on the JVM, where it genuinely turns up — and why the Groovy in your Jenkinsfile is not quite it.

Groovy everywhere, with one large asterisk

Why the nightly check moved to Python, what PowerShell still does better than anything else, and where the seam between the two belongs.

Two hundred lines of PowerShell I could not test

Linux is my default for portable .NET services. Windows containers earn their place with specific dependencies, and a headless Server Core VM still solves problems that a container cannot.

Windows containers, Linux containers, and when a small Windows VM wins

A nightly drift check in Declarative Pipeline and PowerShell: why H in a cron line, exit codes that actually fail a build, and the interpolation bug that cost me an evening.

The scheduled job on the Windows agent

Two days after getting Spark running locally I pointed it at S3, and everything that broke was named org.apache.hadoop. What Hadoop actually is, which third of it Spark replaced, and why committing a write to object storage is still the interesting problem.

I wrote a Spark job. The stack trace was Hadoop.

EMR Serverless went into preview at re:Invent, so I finally sat down with Apache Spark. Notes from the first fortnight: why nothing runs until something has to return, why the partition is the unit of parallelism, and the default of 200 that nobody chose.

The cluster went away. The shuffle did not.

Two VMs, three launch methods that are traps, and the plugin list I actually needed. Everything that happens before the first pipeline runs.

A Windows agent for Jenkins, from bare VM to online

A pricing rule, a few records, and a thin HTTP handler. C# works well with functions and explicit data, without a service interface for every calculation.

C# without the class hierarchy

Forty-six weekend commits later: the two-sources-of-truth problem, the race I only found by reading the code, and the rule I got wrong twice.

A year of automating a flat

Minimal APIs, a smaller startup file, and C# 10 remove useful amounts of ceremony. A complete availability service shows what is easier—and what still needs explicit decisions.

.NET 6 makes small services simpler

I deployed the flat with SAM and then rebuilt it in Pulumi, on purpose, to find out where each one actually earns its keep.

The same stack, written twice

Amazon MemoryDB went GA on the nineteenth of August: Redis 6.2 with a multi-AZ transaction log in front of it. What that costs on the write path, why the replica endpoint became a correctness decision, and why I provisioned it with the CLI.

Redis you are allowed to lose, and Redis you are not

The goodnight routine needed to pause in the middle. That single requirement is what moved it out of Lambda and into Step Functions.

A ten minute wait should not cost you Lambda time

Both of them write CloudFormation for you. Two weekends porting the stack from one to the other, and the difference turned out to be IAM.

SAM and Serverless Framework, on the same flat

SAM is a preprocessor, so the honest way to price it is to print what it hands CloudFormation. Mine is three times the file I typed.

The template I wrote and the one AWS ran

Cron expressions are fixed and sunset is not. A note on solving that with a polling schedule and forty lines of trigonometry.

EventBridge cannot tell you when the sun sets

A complete multi-stage build, an unprivileged process, and a repeatable memory experiment. Image size, managed heap, and container memory are different numbers.

Running .NET 5 in Linux containers—and measuring its memory

Records, init-only properties, and ordinary functions make small C# programs easier to read. The runtime improvements are welcome too, but they deserve measurements of their own.

C# 9 and .NET 5: less ceremony, useful improvements

What Alexa actually sends a Lambda function, and why the discovery response is the part that will bite you.

A smart home skill is a router and a contract

vRealize Automation 7 and 8 were not the same product with a new version number, and every customer engagement started by counting the damage by hand. What we built instead, and why the whole stack was decided by one constraint: it had to run inside a network we were never allowed to leave.

The migration tool that could not phone home

I became a team lead at 23 and assumed my own effort and enthusiasm were the motivation. When the team was not committed, I quietly finished their tasks myself. An itemised account of what that cost.

I build entire products end to end

What I do

Experience

How I work

Recent writing

The check-in app is one web page and two wrappers

A year with Codex, a week with Astra

Onboarding took two days. Now it takes two hours.

Working on something interesting?