![\"A](\"https://jonno.nz/img/posts/the-holes-that-kill-you-are-the-ones-you-never-tested-fig-1.png\")
Auckland Light Rail ran for six years, spent $228 million, and laid zero metres of track.
\nAt its peak the project was paying about $920,000 a week to two engineering firms. Not to build anything. To plan, re-plan, and re-plan again, through three different versions of the route. In January 2024 a new government cancelled it as part of a 100-day plan, and the public got nothing back for a quarter of a billion dollars except some very expensive PDFs.
\nI want to be upfront about what this series is, because the material is political and the timing is an election year. This isn't a political hit job. It's pattern recognition on the public record, and the pattern has both parties' fingerprints all over it.
\nI've spent a little bit of time working with exec teams, at startups and at big companies, and I've seen my share of strategy rewrites. Even one rewrite, done inside one company, with everyone trying their best, costs you about a year. Roadmaps get binned. Half-finished work gets written off. Your best people quietly update their CVs. The new strategy is usually about 70% the old strategy with new diagrams.
\nNew Zealand runs like a company that's contractually required to put its entire exec team up for replacement every three years. Each incoming team arrives with a mandate to show movement in a hundred days, and the fastest way to show movement is to bin whatever the last team was building. The reorg is the announcement. The announcement is the work.
\nAny one of these execs might be competent on their own. The dysfunction is the team, and the team is permanent: it spans parties, decades and ideologies, and it cannot leave a predecessor's decision alone. Meanwhile the shareholders of NZ Inc can't sell their shares. A decent chunk of them are still at primary school.
\nThat's the frame. Now the receipts.
\nThe Cook Strait ferries are the cleanest one. In 2021 KiwiRail signed a fixed-price contract for two new rail-capable ferries, with port upgrades to match. By December 2023 the cost had escalated badly and the incoming government cancelled the lot. By then $507.3 million had already been spent, and 1News later confirmed we paid Hyundai another $144 million to walk away from the shipbuilding contract. So: $651 million, no ferries. Then we ordered different ferries anyway, for $1.86 billion, due 2029. The old rail-enabled Aratere got retired in the meantime, which severed the Cook Strait rail link for roughly five years.
\nWe paid $144 million to not get ships, and then bought ships.
\nThree Waters took seven years from the Havelock North outbreak to a full legislative framework, then got repealed under urgency in an afternoon in February 2024. You can think the reform was wrong and still notice the maths: the $120 to $185 billion of ageing pipes that justified it didn't get repealed with it.
\nThe Resource Management Act replacement is my favourite, in the way a slow-motion replay of a faceplant is someone's favourite. Parliament spent years building the RMA's successor, passed it in August 2023, and the next government repealed it 122 days later. The replacement's replacement arrived in late 2025, and The Spinoff noted its principles "all but mirror" the laws that got repealed. Thirty-plus years of everyone agreeing the RMA must go, and the RMA is still here, having outlived its own successor.
\nTe Pūkenga merged sixteen polytechnics into one national institute in 2020. In January 2026 it finished un-merging them back into ten. Between those two restructures: 855 staff gone, $9.5 million in redundancy payouts, and $325 million allocated to stand up the new polytechs that look a lot like the old polytechs. If you've ever survived a corporate restructure followed by a de-restructure, you know exactly what those six years felt like from inside.
\nThe Smokefree generation law passed in December 2022 as a world first, got studied by half the planet, and was repealed before a single clause took effect.
\nAnd if you want the purest specimen of the whole genre, it's the bright-line test. National invented it in 2015 at two years. Labour stretched it to five, then ten. National snapped it back to two in 2024. Four settings in nine years, on a tax rule that decides what happens when ordinary people sell a house. Both teams, same dial.
\nWorth noting who started and stopped each of these. Labour built ALR, iReX, Three Waters and Te Pūkenga; National stopped them. National invented the bright-line test; Labour extended it; National reverted it. And Labour announced a $785 million harbour cycle bridge in June 2021, then cancelled it itself by October, which proves you don't even need to change the government to get the U-turn. The cycle doesn't care who's driving.
\nTime to add it up, and this is where I need to be careful, because the temptation with a topic like this is to quote the scariest number available.
\nPoliticians on every side do exactly that. When light rail was cancelled, ministers cited builds costing "$15 billion, rising to $29.2 billion". Those are projections of money we now won't spend. Counting avoided future costs as waste is fantasy accounting, and the moment you do it, anyone with a calculator can dismiss your whole argument.
\nSo the rules for this series are boring on purpose. Count money that actually left the public account on things that were then cancelled or reversed: sunk planning, design, contracts, land, plus the penalties paid to walk away. Never count "would have cost" figures. Attribute anything contested to whoever claims it, like Labour's Tangi Utikere putting the full iReX cancellation at $1.16 billion once you include ongoing maintenance, a figure that bundles in things I won't count.
\nOn those rules: light rail's $228 million, iReX's $507.3 million plus the $144 million exit, the taxpayer share of Let's Get Wellington Moving ($109.7 million of the $180.7 million it spent over nine years, mostly on consultants; Wellington's ratepayers carried the rest), Te Pūkenga's redundancy bill, and the $80 million-plus in public service redundancy payouts across 24 agencies as the workforce swung up by 13,000 and back down again.
\nThat's well over a billion dollars in directly sunk and cancellation costs from a single change of government, counted conservatively. A billion dollars is roughly half a new Dunedin Hospital, a project which has itself been rescoped so many times that 35,000 people marched down George Street about it.
\nI lived on the North Shore through the announcement years, and the ritual became familiar: the render, the press conference at the empty site, the revised render, the quiet line in a later Budget. SkyPath, then the standalone cycle bridge, then neither. Light rail to the airport in three different flavours. Somewhere around 2021 I stopped believing artists' impressions the way I'd stopped believing crypto whitepapers.
\nHere's the backdrop that turns this from annoying into expensive. Te Waihanga's 2022 stocktake found that from 2010 to 2019 New Zealand was the biggest infrastructure investor in the OECD as a share of GDP, while sitting near the bottom 10% of high-income countries for how much infrastructure each dollar buys. We're not under-spending. We're spending like a rich country and delivering like a country that re-litigates its decisions every electoral cycle, into an infrastructure deficit Te Waihanga and Sense Partners size at somewhere between $100 and $210 billion over 30 years.
\nI grew up in Christchurch, and the rebuild taught me what both versions of this look like on the ground. The bits that got locked in and left alone got built; the city got a genuinely world-class convention centre and a repaired arts centre. The bits that kept getting re-litigated dragged on so long they became civic jokes. The stadium took fifteen years of arguments and only opened this year. Same city, same decade, same money. The difference was whether each new set of decision-makers honoured the last set's call.
\nA fair pushback: sometimes cancelling a project is the competent move, and an exec team that can't kill anything is its own kind of disaster.
\nTrue, and the record backs it in places. Treasury papers showed about 80% of the iReX cost escalation was seismic strengthening at the ports, work that's needed no matter whose ferries dock there. Pulling the pin on a runaway project can be governance working, and I'd rather have a government willing to stop a bad bet than one that rides it down out of pride.
\nThe target of this series is narrower: the re-litigation of settled, long-horizon decisions. Ferries wear out on a schedule that doesn't care about coalition agreements. Pipes corrode on chemistry's timetable, not the electoral one. When the underlying problem has a 30-year horizon and the decision-making has a 3-year one, reversal isn't prudence. It's the exec team marking its territory, and the bill lands on shareholders who can't vote yet.
\nSo that's Part 1: the cost, counted conservatively, with the fantasy numbers left out.
\nPart 2 asks why New Zealand specifically does this more than nearly anyone, because it turns out we're rigged for it in ways most democracies aren't, and the man who wrote the book on it in 1979 has updated his diagnosis.
\nPart 3 is where I stop reading the receipts and start pulling them. The entire statute book is published as open data: every act, every amendment, every repeal. I've started parsing all of it, back to 2008, to measure the churn properly. It's a personal project and a few evenings of code, not a royal commission. But nobody else has built it, most of NZ Inc's shareholders haven't been born yet, and somebody should be keeping the receipts.
\n","date_published":"Fri, 12 Jun 2026 00:00:00 GMT","image":"https://jonno.nz/og/zero-metres-of-track.png"},{"id":"https://jonno.nz/posts/the-holes-that-kill-you-are-the-ones-you-never-tested/","url":"https://jonno.nz/posts/the-holes-that-kill-you-are-the-ones-you-never-tested/","title":"The holes that kill you are the ones you never tested","content_html":"Every time something goes down, the first instinct in the room is to add a\nlayer. Another replica. A second region. One more health check in front of the\nthing that broke. It feels like progress, and the Swiss cheese model hands that\ninstinct a lovely picture to point at.
\nYou know the diagram even if you've never heard the name. Stacked slices of\ncheese, each slice a layer of defence, every slice riddled with holes. An\naccident only makes it through when the holes in all the slices happen to line\nup. James Reason built it to\nexplain how hospitals and aircraft fail, and it's still the best tool we have\nfor explaining why a system with five safety nets can still face-plant.
\nThe picture has a side effect though. It teaches you to count slices. And\ncounting slices is mostly the wrong job.
\nTake redundancy, since that's where slice-counting does the most damage. The\nmodel says two of everything beats one. Two servers, two regions. That holds\nright up until both copies share a single reason to die at the same instant.
\nOn 19 July 2024 CrowdStrike\nshipped a content update\nto its Falcon sensor and bricked around 8.5 million Windows machines, by\nMicrosoft's estimate, inside a few hours. Every one of those hosts was\n"redundant" in somebody's architecture diagram. Made no difference. They ran the\nsame agent and swallowed the same bad file at the same moment, so they all\ndropped together. When the failure is perfectly correlated like that, redundancy\nbuys you nothing. NASA's reliability folks put it bluntly: if one fault can take\nout the backup too, two subsystems fail twice as often as one.
\n
There's a quieter version of the same trap hiding in your SLOs. String together\na dozen services that are each up 99.9% of the time and your ceiling is about\n98.8% before you've written a line of your own code, because every dependency is\none more slice with its own holes. You can't promise more reliability than the\nflakiest thing you lean on, and each extra nine costs more than the last.
\nThe slices you can draw on a whiteboard are the safe ones. The holes that get\nyou are the ones nobody can see: the failover that's never been run, the\nassumption that stopped being true six months ago when no one was looking, the\ndead code still sitting in production behind a flag.
\nKnight Capital is the one I'd tattoo on a junior engineer. On 1 August 2012 they\nswitched on new trading code and pushed it to seven of their eight servers. On\nthe eighth, a reused flag woke a dead function called Power Peg that should have\nbeen ripped out years earlier. For about 45 minutes it hurled orders into the\nmarket with nothing counting the fills:\nmore than four million executions and 397 million shares.\nKnight's own books put the loss at roughly $440 million, and the firm didn't\nsurvive the week.
\nPull it apart and every hole was survivable on its own: dead code left in the\nbuild, a flag doing double duty, a deploy one person ran with nobody reviewing\nit, 97 warning emails before the open that everyone read as noise. None of those\nsinks you alone. Line them up on a single Tuesday morning and the company is\ngone.
\nThat's Reason's real point, and it's a good one. The model is right. It's just\ncoarse. Richard Cook said it better in\nHow Complex Systems Fail: complex systems\nrun in a degraded state the whole time, stuffed with small faults, staying up\nbecause people quietly hold them together. Catastrophe needs a few of those\nfaults to meet. A sixth slice does nothing about the holes already living in the\nfive you've got.
\nYour architecture diagram and your runbook describe the system you imagine you\nhave. The real system is whatever your on-call engineer does at 3am to keep it\nlimping along. Erik Hollnagel calls that gap\nwork-as-imagined versus work-as-done,\nand the holes love living in it.
\nThe useful work is dragging those invisible holes into daylight while you still\nget to pick the timing. Adrian Cockcroft has the perfect name for skipping it.\nHe calls an untested failover\n"availability theatre":\nyou've got the runbook, you've got the standby, you feel great, and you have no\nclue whether any of it works because you've never once pulled the plug to watch.
\n![\"A](\"https://jonno.nz/img/posts/the-holes-that-kill-you-are-the-ones-you-never-tested-fig-2.png\")
This is why the teams who are good at this spend their hours on stuff that looks\nunglamorous. They run game days and break things in production on purpose, the\nway Netflix set Chaos Monkey loose to kill its own servers at random, just to\nprove the system could take it. They write\nblameless postmortems,\nbecause the day people get punished for an outage is the day they stop telling\nyou where the holes are. They treat reliability as a budget they spend, not a\nfeeling they chase.
\nMost of that is a people problem wearing a technology costume. The\nhighest-leverage slice in your whole stack is usually the on-call engineer at\n3am, and whether your culture lets them say "yeah, I broke it, here's how"\nwithout flinching. No cloud provider sells that one.
\nI've built billing and payments platforms where downtime means somebody doesn't\nget paid, and shipped to production many times a day. The urge to bolt on\nanother layer never goes away. It just gets more expensive, and more comforting.
\nBy all means keep your redundancy. Keep the regions and the replicas. Just don't\nkid yourself that a standby you've never failed over to is a second slice of\ncheese. It's a hole with a comforting label, and you'll find out which one it\nreally is on the worst possible morning.
\n","date_published":"Sun, 31 May 2026 00:00:00 GMT","image":"https://jonno.nz/og/the-holes-that-kill-you-are-the-ones-you-never-tested.png"},{"id":"https://jonno.nz/posts/what-happens-when-a-worm-drives-claude/","url":"https://jonno.nz/posts/what-happens-when-a-worm-drives-claude/","title":"What Happens When a Worm Drives Claude?","content_html":"A few weeks ago I was on the couch with my eight-year-old, and she was going on\nabout worms. Kids ask the kind of questions we forget to ask as adults, the ones\nthat drop off once we get all mature and sensible. That conversation is the\nreason a worm ended up driving Claude.
\nRight now everyone builds LLM agents the same way. Take a model, bolt more tools\nonto it, hand it a bigger toolbox. I wanted to try the opposite. Leave the model\nalone, and put a brain inside the loop to do the steering.
\nI'll say this up front: it's a fun project, not a serious one. But the thing it\ndoes is genuinely strange, so it's worth a watch.
\n(Can't see the embed? Watch it on YouTube.) All\nthe code is on GitHub if you'd rather pull\nit apart yourself.
\nThink of it as a car. Claude (Sonnet 4) is the engine. A big engine with nobody\nat the wheel. It has five tools: read, grep, write, bash, and run the tests.\nThat's the lot.
\nThe driver is a small controller program. Every tick it looks at what Claude\njust did and decides what Claude does next. It never writes a line of code\nitself. It just shapes the behaviour, the way you'd drive a car without ever\ntouching the pistons.
\nAnd the driver doesn't read the code either. It reads the dashboard. Five\nnumbers, every tick, and only these five: how many tests are failing, the share\npassing, how many files Claude has touched, how many ticks have gone by, and the\nlast tool it used. That's the whole view of the world.
\nOut the other side there are seven knobs. The big one is the gear: diagnose,\nedit, test, or stop. Then a handful more, like how much risk to take, how long\nto think, how widely to read, how hard to commit, and how done it reckons it is.\nFive numbers in, seven knobs out.
\nThe worm.
\nIt's C. elegans, a roundworm about a millimetre long. It has 302 neurons, and\nscientists mapped every single connection between them back in 1986 (the year I\nwas born, which I'm choosing not to read anything into). It's one of the only\nnervous systems we understand all the way down.
\nI took 14 of those neurons and ran them live in a simulation. Four of them do\nthe heavy lifting. One is a salt sensor, and to a worm salt means food, so I\nwired it to reward. Its opposite I wired to bad results. One drives the worm\nforward, so I wired it to how much work is left. One throws it into reverse, so\nI wired it to errors.
\nI didn't make any of that wiring up. It's the published connectome. All I did\nwas connect the worm's senses to the agent's situation and let it run.
\n(If you want the longer argument for why a worm of all things, I made the case\nwhen I\nfirst started this.)
\nDoes it crash, or does it do something useful?
\nIt starts at baseline with one failing test. Claude opens a few files, runs a\nsearch. The tests are failing, so the worm leans on the accelerator. I never\nwrote a rule that says "press the accelerator when there's work to do". It falls\nstraight out of the wiring.
\nThen Claude runs the tests and gets a real failure. The error neuron lights up,\nthe worm changes gear, and Claude makes its first proper edit. Risk drops, it\nreads less, it commits harder. Again, no rule for that combination. The error\nsignal and the work-left signal crossed a threshold together, and the worm\nturned that into "stop wandering, make a change". It found a scent.
\nA test goes green. Then another. The worm is cooking. If the run ended there I'd\nhave gone home happy.
\nIt didn't. The second edit was sloppy and broke something else. The pass rate\ndrops, the errors jump, and the punish neuron (my favourite, for the record)\nlights up hot.
\nThis is the moment I cared about. The obvious move is to panic, drop back to\nsquare one, and start over. The worm doesn't. It stays in gear, eases off, gets\ncareful, reads before it edits. Two ticks later the careful edit lands and the\npass rate climbs back higher than it was before the break.
\nIt recovered without restarting, which is exactly what a good pair would do.\nLook at the diff, don't burn the house down and start fresh. And nothing in the\nwiring connects a regression to a change of gear. The punish signal pulls one\nknob down. It doesn't yank the worm out of edit mode, because that's a different\nknob entirely.
\nBy tick 14 the last test goes green, ten out of ten, and the worm stops and\nparks the car. Not because I told it to stop at ten, but because there was\nnothing left pulling it forward.
\nSo I ran it properly. About 200 times. The live worm averaged a 0.96 pass rate.
\nThen I ran the exact same connectome, same wiring, same neurons, but\npre-recorded, like playing a tape of the worm's brain instead of letting it\nreact. That version averaged 0.87, and the live worm beat it every single time.\nSame brain. The only difference is one of them was alive and getting feedback,\nand the other was just replaying.
\nNow the part I have to be honest about. I ran a colder version where Claude\nhadn't indexed the repo first, and plain Claude with no worm and no driver beat\nevery controller I built, the live worm included. The worm is great at following\na gradient once the ground is mapped. It cannot plan its way around a codebase\nit has never seen. So this isn't "worm beats Claude". It's "feedback beats no\nfeedback", which is a smaller and far more useful claim. (The full methodology\nand numbers are\nin the paper.)
\nGive your agent a way to know when it's done. The baseline that couldn't tell\nkept running long after the job was finished, burning money for nothing.
\nDon't restart on a setback. The worm that held its nerve through the regression\ndid most of the real work.
\nAnd feedback beats wiring. Two identical brains, and the live one won purely\nbecause it was in the loop. The data you feed an agent and the loop you close\naround it matter more than how clever the thing in the middle is.
\nI built this in about three weeks, a couple of evenings here and there, on\nroughly $50 of API credits. I used Claude Code as my research assistant, which\nmeans my brain was steering Claude to build a thing that lets a worm's brain\nsteer Claude.
\nA few years ago a question this daft would have been a research project. Now\nit's a weekend and a coffee budget. The cost of chasing a strange idea has\nfallen through the floor.
\nSo chase the strange ideas. The code's\non GitHub. Go on.
\n","date_published":"Sun, 31 May 2026 00:00:00 GMT","image":"https://jonno.nz/og/what-happens-when-a-worm-drives-claude.png"},{"id":"https://jonno.nz/posts/the-18-laws-of-human-nature/","url":"https://jonno.nz/posts/the-18-laws-of-human-nature/","title":"The Laws of Human Nature","content_html":"\nGreene's order: master yourself, then decode others, then handle the social dynamics. Builds the vocabulary you need for the later laws.
\nThe five that change the most about how you read a room. Irrationality, role-playing, envy, defensiveness, aimlessness.
\nGreene at his most surgical. Repression, grandiosity, envy, death denial. Read these slowly and with someone who'll tell you the truth.
\nI recently finished Robert Greene's The Laws of Human Nature and it wouldn't\nleave me alone. Eighteen laws, each one a separate pattern in how people\nactually behave (not how we like to think we behave). I kept catching myself\nwatching the patterns play out in real life: in meetings, in news cycles, in my\nown head.
\nSo I built the thing above. Eighteen cards, one per law. Tap any of them to read\nmy full take, with two real-world examples and a short list of behavioural\nthings you can do this week. About a thousand words a law, around twenty\nthousand total. Your read-progress sticks between visits.
\nIf a card grabs you, that's the one to start with. If not, the canonical order\nat the top is a fine path. Either way, the actual book is much richer than this\nmap. Robert Greene wrote it; if you want the deep version,\ngrab a copy on Amazon{target="_blank"\nrel="noopener"}.
\nA note on attribution. The names, the structure and the underlying observations\nare all Greene's (The Laws of Human Nature, Profile Books, 2018). The writing\nis mine, the modern examples are mine, and any wrong reading of his ideas is\nmine.
\n\n\n\n\n\n\n\n","date_published":"Tue, 26 May 2026 12:00:00 GMT","image":"https://jonno.nz/og/the-18-laws-of-human-nature.png"},{"id":"https://jonno.nz/posts/conscious-minimalism/","url":"https://jonno.nz/posts/conscious-minimalism/","title":"Conscious Minimalism","content_html":"I build with Conscious Minimalism.
\nConscious Minimalism means every decision, every feature, every commitment is a\nchoice. Nothing ships by default. The shape grows with the customer, not the\nroadmap.
\nI obsess over the interface, not the internals. The boundary is what you live\nwith, and what you refactor and scale.
\nI optimise for speed to de-validation. The faster I kill a bad idea, the less\nsunk cost owns me.
\nI fix one narrow painpoint, not a broad category or a platform play. One thing,\ndone properly.
\nI aim to be the best at that one feature, not one of the good ones in the list.
\nI chase niche over crowded. Niche means the idea is more unique, with a better\noutcome than the legacy approach.
\nCrowded is a signal of competition. Without a 10x better solution, you probably\nwill not make it.
\nI bet on AI-native platforms and APIs. Anything that does not use AI as a\nprimitive is already legacy.
\n","date_published":"Thu, 14 May 2026 12:00:00 GMT","image":"https://jonno.nz/og/conscious-minimalism.png"},{"id":"https://jonno.nz/posts/the-dent-and-the-crater/","url":"https://jonno.nz/posts/the-dent-and-the-crater/","title":"The dent and the crater","content_html":"\n\n"The world breaks every one and afterward many are strong at the broken\nplaces. But those that will not break it kills."
\n— Ernest Hemingway, A Farewell to Arms (1929)
\n
Hemingway wasn't writing about leadership but he might as well have been. Some\nof the senior leaders I've worked with have taken serious dents over their\ncareers. The board that turns. The launch that lands flat. The cofounder who\nwalks. The round that doesn't close. The dents come with the job. The question\nis what's there a year later: a place you went back to and got strong at, or a\ncrater you've been quietly walking around since.
\nThe crater is the part that doesn't make the leadership playbooks. The bad year\ndoesn't get you on its own. The silent processing of the experience afterwards\ndoes, the quiet work of making sure that bad year can't happen again. You start\nshowing up to meetings already half-armoured. You stop trusting the kind of\ndecision that hurt you the last time. You add a layer of process around the work\nbecause process is what you wish you'd had then. Most of it doesn't land as a\nwound on the day. By the time it sets in, you've already started telling\nyourself a different story about it: you've learned, you've matured, you're\noperating at a higher level now.
\nThat's the bit that makes it nasty. Hardening reads as wisdom from the inside.\nYou flinch at the next pitch and call it prudence. The new hire brings an idea\nyou don't like, so you tell yourself you've seen this movie before. You shut\nyour door and call it focus. The mind narrates the armour as growth more often\nthan not, and the people around you go along with it because senior leaders\naren't expected to second-guess themselves out loud.
\nThe cost shows up sideways. You don't notice the day you stopped being\npersuadable by a junior engineer. You notice five years later, when the people\nworking for you have stopped bringing you the rough version of an idea because\nthey know what you'll do with it. The day you got bored of customer calls is the\nsame. You feel it a quarter later, when the deck you wrote turns out to be three\ncalls behind reality. The dents you didn't tend become the things you stop being\nable to see. Walls have that habit. They keep the threat out and the signal out\nat the same rate.
\nThe leaders who stay good at this over time do something specific about it. They\ndidn't get there by accident. Some have a coach they actually use, not the one\nHR signed them up with. A few run a peer group of two or three other operators\nwhere the deal is you say the real thing about the quarter, including the bit\nyou're embarrassed about. Others lean on a partner or a mate outside the\nindustry who couldn't care less what your title is and wouldn't be impressed if\nyou told them. The best ones I've watched have a practice that takes them\noff-stage. Riding, surfing, building, training. Doesn't matter what. The point\nis having a part of life where you're not on stage.
\nThe shape underneath these is the same. They keep at least one room in their\nlife where the armour comes off, and they go in there often enough that the dent\ndoesn't get to set. That's the work. It looks soft from outside. It isn't. It's\nmost of the reason these people are still doing the job at fifty in a way that\nresembles how they did it at thirty.
\nPeople who skip this are the senior leaders you've been managed by at some point\nin your career, the ones you probably promised yourself you wouldn't become.\nThey've stopped reading the room. Most new ideas look like traps. Their\ncalendars are locked down to keep surprises out. They're not bad people. They're\npeople whose dents won, and the orgs around them are downstream of that win in\nways few people have the standing to call out.
\nThe team picks it up. That's the part founders and senior people miss most. You\ndon't get to decide which of your behaviours your reports model and which they\nignore. They copy the closed-off ones at the same rate as the rest, because\nyou're the senior person and that's the working definition of culture in\npractice. Your defensiveness teaches them caution. Your shortened patience\nteaches them to close things down faster than they should. The day you refused\nto say "I don't know" is the day they stopped saying it too. The crater spreads\noutward into the org and stops being something you can fix in private.
\nTending the dent is leadership work, the same line item as strategy, hiring, and\nboard management. Skip it for long enough and the dent becomes a crater, the\ncrater becomes the shape of the org, and the people who used to bring you their\nbest ideas start bringing them to someone else.
\n","date_published":"Wed, 06 May 2026 12:00:00 GMT","image":"https://jonno.nz/og/the-dent-and-the-crater.png"},{"id":"https://jonno.nz/posts/built-a-read-later-chrome-extension-because-pocket-died/","url":"https://jonno.nz/posts/built-a-read-later-chrome-extension-because-pocket-died/","title":"I Built a Read-Later Chrome Extension Because Pocket Died","content_html":"I open about thirty tabs a day. I read maybe three of them. The rest are "oh\nthat looks interesting, I'll come back to that," and then they sit there until\nmy browser starts choking and I close everything in a fit of guilt.
\nBookmarks aren't the answer. Bookmarks go into a folder I never open. Pocket was\nthe answer, except\nMozilla shut Pocket down in July 2025,\nwhich I found out the way most people did: by going to save something and\ndiscovering the service was gone.
\nSo I built\nRead Later.\nIt's a Chrome extension. It saves the page you're on. That's the whole pitch.
\nYou hit Cmd+Shift+L and the current tab gets saved with its title, URL, and\nfavicon. Add a tag if you want. Done. There's a popup if you'd rather click a\nbutton, and a context menu if you'd rather right-click a link without opening it\nfirst.
There's also a full-tab "shelf" view at Cmd+Shift+K where all your saved\narticles live. Search, filter by tag, mark as read, archive. The aesthetic is\nwarm parchment and moss green because I got tired of every productivity tool\nlooking like a Linear screenshot.
That's it. No AI summaries, no recommendations, no "people who saved this also\nsaved...", no newsletter, no account.
\nI tried. Bookmarks are designed for things you'll come back to many times. A\nread-later list is the opposite, you read each thing once and then it's done.\nThe lifecycle is different. Stuffing them into the same UI is why my bookmarks\nbar has been a graveyard for ten years.
\nThe other read-later apps that survived Pocket all want an account, a sync\nserver, and usually a subscription. For something whose whole job is "remember\nthis URL until I read it," that's a lot of infrastructure. I wanted local\nstorage and nothing else.
\nEverything lives in chrome.storage.local. Nothing leaves your browser. There's\nno server because there's nothing for a server to do. If you want to back up\nyour list, there's an Export button that gives you JSON. If you want to move it\nto another machine, Import takes the JSON back. If you want to share your shelf\nwith someone, "Copy as Markdown" gives you a clean list you can paste into Slack\nor an email.
This was a deliberate call. Sync is the feature that turns a small tool into a\nservice, and a service needs accounts, and accounts need a backend, and a\nbackend needs my time and money forever. JSON in, JSON out is the version of\n"sync" that costs me nothing and gives the user complete control. Want it on two\nmachines? Export on one, Import on the other. Good enough.
\nThe real experiment was the launch. I've shipped plenty of things over the years\nbut never to the Chrome Web Store, and I wanted to see what that pipeline felt\nlike end to end. Pretty smooth, no drama. You pay the developer fee, fill in the\nlisting, upload a zipped manifest, wait for review. Mine went through first\ntime.
\nThe work was in the listing copy and the screenshots. You're suddenly writing\nfor a discovery page where people decide in three seconds whether to click\ninstall. That's a different muscle from writing a README. It made me think\nharder about what the extension actually does for someone who isn't me.
\nI've got a folder on my laptop called "side projects" with about a dozen things\nin it, each solving some specific bit of friction in my own life. Most of them\nstay there. This one made it out because the friction was real, the build took\nan afternoon, and putting something on the Chrome Web Store turned out to be a\nsatisfying little exercise.
\nIf you've got tabs piling up and don't want to hand your reading list to a\nstartup, it's\nthere.\nSource on GitHub, MIT licensed. Pin\nthe leaf icon and have a go.
\n","date_published":"Tue, 05 May 2026 12:00:00 GMT","image":"https://jonno.nz/og/built-a-read-later-chrome-extension-because-pocket-died.png"},{"id":"https://jonno.nz/posts/product-market-fit-is-a-gauntlet/","url":"https://jonno.nz/posts/product-market-fit-is-a-gauntlet/","title":"Product market fit isn't a stage, it's a gauntlet","content_html":"Product market fit gets sold as a milestone. Find it and you're off to the\nraces. That's the bit nobody who's been through it actually believes.
\nPMF is a gauntlet. It eats teams, it bends founders, and it quietly poisons the\ntechnical decisions you're proud of at the time. Most of the damage I've watched\ndone to good companies wasn't from missing PMF. It was from how they behaved\nwhile they were looking for it.
\n![\"The](\"https://jonno.nz/img/posts/pmf-gauntlet/gauntlet-loop.svg\")
There's a particular kind of person who does well in the pre-PMF phase. High\ntolerance for ambiguity, low need for closure. The deck never feels finished,\nthe metric you're chasing changes every six weeks, and the answer to "what are\nwe doing in three months" is "depends."
\nPlenty of really good operators just cannot function in that environment. That's\nnot a character flaw — it's a stage mismatch. Some people thrive at zero to one.\nSome thrive at one to ten. Almost no one thrives at both, and the industry\npretending otherwise has cost a lot of careers and a lot of sanity.
\nNaming this honestly so people can self-select is one of the kindest things a\nfounder can do. You're not letting someone down by saying "this stage probably\nisn't for you, but the next one will be." You're saving them eighteen months of\nfeeling broken.
\nTiming, market, economy, what your one big regulator decides on a Tuesday — none\nof that is yours. You can have a sharp thesis and a great team and ship\nsomething nobody buys, because something three layers above you shifted while\nyou were heads-down.
\nThe only protection I've found against this is a vision the team is genuinely\nbought into. Not the slide. Not the wall poster. The actual reason you all got\nout of bed this morning. When the macro turns and the metric you were proud of\nlast quarter goes sideways, that vision is what stops the org from devouring\nitself.
\nI've watched startups where the thesis was right but the timing was a year early\nlose half their team in three months because nobody could explain why they were\nstill doing what they were doing. It wasn't a strategy problem. It was an\nalignment problem dressed up as a strategy problem.
\nThis is also where founders take the most damage personally. You can do\neverything well and still get hit by something nobody could have predicted. If\nyour sense of self is tied to PMF being a verdict on you, that breaks people.\nThe ones I've seen come through it healthy treated PMF like weather they were\nnavigating, not a test they were passing.
\nYour moat isn't the product. It isn't the tech. It definitely isn't the brand.\nYour moat is how fast the org can spot a shift in TAM or target market and\ntranslate it into a product move.
\nDays, ideally. Weeks if you have to. Not quarters.
\nThis is where the technical decisions made in the name of "scaling" quietly\ncripple you. The microservices you split out before you needed to. The custom\ninfrastructure someone stood up because their last job had it. The platform\nabstractions that mean a small UI change touches four repos. Each of those felt\ndisciplined at the time. Each is now a tax on the only thing you actually have —\nspeed.
\nAndreessen wrote the\noriginal PMF essay almost\ntwenty years ago, and the line that's aged best is the bit about doing whatever\nit takes — changing people, rewriting the product, moving markets. That's not a\nlicense to be chaotic. It's a reminder that the org needs to be physically\ncapable of those moves. If your architecture, process, or contracts make\nrewriting the product a six-month project, you've already lost the gauntlet\nwhether you know it yet or not.
\nI've got a strong opinion on this one: when in doubt, build it boring. Boring is\nfast to change.
\nThe customers who signed up first kept you alive. They also signed up for a\nslightly different company than the one you're now trying to become. That gap is\nwhere a lot of startups quietly die.
\nKeep them too happy and you slow your evolution. Push too hard toward the new\nvision and you churn the cohort that's funding your runway. The actual job is to\ndo both at once, which is why I sometimes call it internal schizophrenia. You're\na different company to them than you are to yourselves, and that's not a bug —\nthat's the mode you're operating in.
\nThe skill is being honest with the early cohort about where you're going without\nselling them something they didn't buy. The art is using their feedback to\nsharpen the bigger vision rather than letting yourself be pulled back into being\ntheir bespoke vendor. The dance is doing both of those without your team\nthinking you've gone off-piste, because the gap between "what we're shipping\ntoday" and "where we're going" looks weird from the inside.
\nThree patterns I keep seeing.
\n![\"Discipline](\"https://jonno.nz/img/posts/pmf-gauntlet/discipline-vs-fragility.svg\")
Engineering over-scales the architecture. The team builds for the company they\nwant to be in two years instead of the company they need to be this quarter. By\nthe time PMF actually shows up, the org can't move. Worse, the engineers feel\nbusy and capable the whole time it's happening, which is why it's so hard to\nstop. Nobody is asking to slow down — everyone is shipping.
\nProduct holds the roadmap too tightly. The roadmap is the experiment at this\nstage. Treating it like a commitment is a category error. The product teams I've\nseen do this well treat the roadmap like a hypothesis with version numbers —\nlast month's was wrong, this month's is less wrong, and that's how it's supposed\nto feel. The ones who don't end up defending decisions they made when they knew\nless.
\nFounder comprehension debt builds up faster than anyone notices. The founder is\nheads-down on signal — every customer call, every dropped deal, every weird\npattern in the data lands in their head and gets metabolised on the spot. The\nteam is two beats behind, working from last week's mental model. Each individual\ndelay feels minor. The cumulative gap is the thing that kills decisions.
\nEach of these looks like discipline from the inside. Each of these is fragility\nwearing discipline's clothes.
\nMoats in the AI space are shifting quarter by quarter right now. Feature moats\nhave basically collapsed — anything you can describe in a screenshot can be\ncloned in a weekend with the current generation of tools. What's\nactually defensible has moved\ntoward proprietary data, deeply embedded workflows, distribution, trust, and\nregulatory positioning.
\nFor a founder in the PMF gauntlet that means the playbook is unreliable in a way\nit wasn't five years ago. You can't just lift what worked for the last cohort of\nSaaS winners and run it. You have to reason from first principles about where\nyour actual edge is going to come from over the next eighteen months, and place\nchips accordingly.
\nThe gauntlet itself hasn't changed. The chips you're placing have. That's harder\nthan it sounds, because most of us were trained in an era when the moat\nconversation was settled.
\nThe version of PMF that gets written up in case studies is always cleaner than\nthe thing itself. There's a chart, a moment, a story arc. From the inside it\nnever looks like that.
\nIt looks like a year of conflicting signals. Two customers who love it, four who\nchurned, one who would buy if you built a feature you're not sure fits the\nvision. A board update where you're trying to make a noisy graph sound coherent.\nAn engineer asking a perfectly reasonable question about the roadmap that you\ncan't answer without lying a little.
\nThe companies that come through it don't seem, in retrospect, to have had better\ninformation than the ones that didn't. They had roughly the same fog. They just\nkept reasoning out loud, kept revising in public, and didn't pretend the chart\nwas already drawn.
\nThe case studies leave the fog out. That's most of it.
\n","date_published":"Fri, 01 May 2026 00:00:00 GMT","image":"https://jonno.nz/og/product-market-fit-is-a-gauntlet.png"},{"id":"https://jonno.nz/posts/change-management/","url":"https://jonno.nz/posts/change-management/","title":"Change management","content_html":"There's a whole business discipline called change management. Frameworks,\ncertifications, consultancies, the lot. Every big company has someone running it\nduring a restructure or a tech migration. Nobody runs it for you when your life\nturns over.
\nWhich is strange, because the personal version is the harder problem — and right\nnow, more people are facing it than at any point in recent memory.
\nMore than\n92,000 tech workers have been laid off in 2026 alone,\nbringing the total close to 900,000 since 2020. Meta cut 8,000 jobs last week.\nMicrosoft offered buyouts to 7% of its US workforce — the first time in its\n51-year history. Oracle has started cuts that could reach 30,000 by year end.\nCloser to home, Xero, Sharesies, Spark, One NZ and Eroad have all run their own\nrounds. AI is the headline reason, but the impact lands the same regardless of\nthe cause: hundreds of thousands of people closing a laptop and discovering\ntheir working identity has just been deleted.
\nThat's a lot of people being handed a forced version of personal change\nmanagement without ever signing up for the course.
\nThe business framing has the right insight buried in it.\nWilliam Bridges made a\ndistinction in the 90s that most people miss: change is external, transition is\ninternal. Change is the new org chart, the redundancy email, the merger.\nTransition is what happens inside people's heads while all that is going on.\nChange can happen overnight. Transition takes as long as it takes.
\nPersonal change management is just transition without the org chart.
\nI've been through a few years of it now. Not one big event — more like a slow\nstack of endings, some chosen, some not. Companies, relationships, versions of\nmyself I'd been building for a decade. The kind of stretch where you don't\nreally notice you're changing until you look up one day and the old you is gone.
\nThat's the part nobody warns you about. Real change isn't transformation. It's a\ncontrolled demolition followed by a slow rebuild, with a long, weird middle bit\nwhere neither the old you nor the new you is really there.
\nThe thing that goes is usually the organising self. Whatever the old you was\narranged around — a fear, a need for approval, a story about who you had to be,\nan ambition that was really a wound. When that goes, the structure it was\nholding up collapses. That's the death. It's real.
\nWhat survives is everything that wasn't load-bearing on the old arrangement.\nYour humour, your curiosity, the way you actually see people, the things you\ngenuinely care about. Those don't die because they weren't propping anything up.\nThey were just you, underneath.
\nThe disorienting part is feeling like a stranger to yourself and entirely\ncontinuous, at the same time. Both are true. The continuous parts are\ncontinuous. The organising self is gone. You're in between.
\nBridges calls this the neutral zone. The old reality has gone, the new one isn't\nthere yet. He says it's the hardest phase to manage, and most organisations rush\nthrough it because it looks unproductive. People do the same thing to\nthemselves.
\nThe temptation is to build a new identity fast, because the empty space is\nuncomfortable. Don't. Whatever you grab in a hurry will be made of whatever was\nlying around — which usually means the old patterns sneak back in wearing new\nclothes. Workaholism becomes "building my legacy". Approval-seeking becomes\n"being of service". Avoidance becomes "protecting my peace". Same machine, new\npaint.
\nThe test is always: is this coming from fear or from truth? You'll know. The\nbody knows before the mind does. Pay attention to the part of you that goes\nquiet around certain people, certain projects, certain decisions. That's the\nsignal.
\nYou don't get to fearless by trying. You get there by going through enough\nendings that the bluff stops working.
\nFear runs on a specific con: if this thing happens, you won't survive it. Not\nliterally die — but the you that exists now won't continue. You'll be broken,\nfinished, unrecognisable. The con works as long as it's untested. Then the thing\nhappens, and you go through it, and on the other side you notice you're still\nhere. Different, scarred, but continuous. The fear was lying about its hand.
\nAfter that, fear can still show up — it doesn't leave — but it can't run the\nsame con. You've seen the card it was holding. Next time it says you won't\nsurvive this, some quiet part of you knows: I already did.
\nThat's not the absence of fear. It's knowing you can act from what's true even\nwith the fear in the room.
\nHere's the bit that surprised me. Once the demolition is done and the rebuild\nstarts, what comes back isn't impressive. It's just real. Less reactive. Less\nnoise. Less performance. You stop needing to be seen a particular way, partly\nbecause you've watched a few of those selves die and you don't trust the next\none enough to stake everything on it.
\nThe goal of change management — the personal kind — isn't to become someone\nadmirable. It's to become someone who's the same alone as in public. Someone who\ndoes the next true thing without announcing it. Most of the depth of this stuff\nlives in the texture of regular days. How you handle a boring Tuesday. Whether\nyou rest when you're tired or push through to prove something to nobody.
\nBusiness change management has all this in it, and most people read it as a\nproject manager's manual. It's also a personal one. Endings, neutral zone, new\nbeginnings. Same shape, different blast radius.
\nThe seeds that grow through the demolition are the ones worth tending. The rest\nsorts itself out.
\n","date_published":"Sat, 25 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/change-management.png"},{"id":"https://jonno.nz/posts/three-ways-to-look-at-time/","url":"https://jonno.nz/posts/three-ways-to-look-at-time/","title":"Three Ways to Look at Time","content_html":"ST-ResNet's core insight is that not all history is created equal.
\nWhen you're predicting crime in Auckland next month, three different kinds of\npast information matter. What happened in the last couple of months: the recent\ntrend. What happened at the same time last year: the seasonal pattern. And\nwhat's been happening over the longer term: whether crime is generally rising or\nfalling in an area.
\nConvLSTM treats all of this as one continuous sequence and hopes the network\nfigures out which parts matter. ST-ResNet\ntakes a more opinionated approach. It separates these three temporal scales\nexplicitly and gives each one its own dedicated neural network branch.
\nThe original paper by Zhang et al. was about predicting crowd flows in Beijing.\nPeople move through cities in patterns that look a lot like crime patterns:\ndaily rhythms, weekly cycles, long-term trends. The architecture\ntranslates well to crime data,\nwith some modifications.
\nThe three branches each look at different slices of history:
\nCloseness captures what's been happening recently. For our monthly data,\nthis means the last 3 months. If South Auckland has been trending upward over\nthe last quarter, the closeness branch sees that momentum.
\nPeriod captures seasonal patterns. It looks at the same month in previous\nyears. So to predict January 2026, it pulls in January 2025 and January 2024.\nThe assumption is that crime has an annual rhythm, and the same month tends to\nlook similar year to year.
\nTrend captures longer-term shifts. It uses quarterly averages from further\nback: broad strokes of whether an area is seeing more or less crime over time.\nThis is the slowest-moving signal.
\nEach branch independently processes its temporal slice through a stack of\nresidual convolutional blocks, then a learned fusion layer combines the three\noutputs:
\nprediction = W_c · closeness + W_p · period + W_t · trend + bias\nWhere W_c, W_p, and W_t are learned weights that vary by grid cell. This\nis a nice touch. It means the model can decide that the CBD's crime is mostly\ndriven by recent trends (closeness), while a residential suburb might be more\nseasonal (period). Different areas get different temporal recipes.
Each branch uses residual convolutional units, the building blocks that made\nResNet so successful in image recognition.
\nThe key idea: instead of learning the full output at each layer, the network\nlearns the residual, the difference between input and output. The identity\nshortcut connection means gradients flow cleanly through the network during\ntraining, which lets you stack more layers without the signal degrading.
\nResUnit(X) = ReLU(Conv(ReLU(Conv(X))) + X)\nThat + X at the end is the skip connection. If the layer has nothing useful to\nadd, it can learn weights near zero and just pass the input through. This makes\ndeeper networks stable, which matters when you're trying to learn spatial\nfeatures at multiple scales.
For our grid, I use 4 residual units per branch. Each unit has two 3×3\nconvolutional layers with 32 filters. That's deep enough to capture spatial\nrelationships across several kilometres without being so deep that the model\noverfits on 36 months of training data.
\nHere's where theory meets reality, and it gets a bit awkward.
\nST-ResNet was designed for dense, high-frequency data. The Beijing crowd flow\npaper used 30-minute intervals over months of data: thousands of timesteps. The\ncrime papers that report strong results typically use daily data over several\nyears.
\nWe have 48 monthly timesteps. Total. The period branch (which looks at the same\nmonth in previous years) has at most 3 data points per month (2022, 2023, 2024\nto predict 2025/2026). The trend branch is working with quarterly averages from\na four-year window. It's not a lot of temporal data for an architecture that's\nspecifically designed to decompose temporal patterns.
\nI had a feeling this would be the bottleneck, and it was.
\nCloseness branch:\n Input: last 3 months (3 × 6 channels = 18 input channels)\n → 4 ResUnits (32 filters, 3×3 kernels)\n → Output: 32 channels\n\nPeriod branch:\n Input: same month from 2 prior years (2 × 6 = 12 input channels)\n → 4 ResUnits (32 filters, 3×3 kernels)\n → Output: 32 channels\n\nTrend branch:\n Input: 2 quarterly averages (2 × 6 = 12 input channels)\n → 4 ResUnits (32 filters, 3×3 kernels)\n → Output: 32 channels\n\nFusion:\n → Learned weighted sum across branches\n → Conv2d(32, 6, 1×1) → 6 crime type predictions\nTotal parameters: roughly 180k. Slightly smaller than the ConvLSTM, which is\nfine. ST-ResNet's power is supposed to come from the temporal decomposition, not\nfrom model size.
\nTraining uses the same setup as ConvLSTM: Adam optimiser, learning rate 1e-4,\nMSE loss on log1p-transformed values, early stopping with patience of 15\nepochs. On CPU, each run takes about 35 minutes, a bit faster than ConvLSTM\nsince there's no sequential recurrence to deal with.
| Crime Type | \nHist. Avg MAE | \nConvLSTM MAE | \nST-ResNet MAE | \n
|---|---|---|---|
| Theft | \n1.28 | \n1.14 | \n1.18 | \n
| Burglary | \n0.35 | \n0.32 | \n0.33 | \n
| Assault | \n0.20 | \n0.19 | \n0.19 | \n
| Robbery | \n0.04 | \n0.04 | \n0.04 | \n
| Sexual | \n0.03 | \n0.03 | \n0.03 | \n
| Harm | \n0.01 | \n0.01 | \n0.01 | \n
| All types | \n0.39 | \n0.35 | \n0.36 | \n
ST-ResNet beats the historical average but doesn't quite match ConvLSTM. The\naggregate MAE of 0.36 is a 7.7% improvement over the baseline, compared to\nConvLSTM's 10.3%.
\nThat's not a terrible result, but it's not what I was hoping for.
\nWhen I dug into the learned fusion weights, the story became clear. The\ncloseness branch dominates. It gets 60–70% of the weight across most grid cells.\nThe period branch gets 20–25%, and the trend branch barely contributes at\n10–15%.
\nThe model is basically saying: "Recent months matter most, seasonal patterns\nhelp a bit, and long-term trends are mostly noise." That's not a failure of the\narchitecture. It's a fair assessment of what's in the data.
\nWith only 2–3 examples of each calendar month, the period branch can't reliably\nlearn seasonal patterns. It's overfitting to individual years rather than\nextracting a stable seasonal signal. ConvLSTM handles this better because it\nprocesses the full sequence and implicitly learns seasonality from the\ncontinuous flow of months, without needing to explicitly align calendar periods.
\nThe trend branch suffers even more. Quarterly averages over a four-year window\ndon't give it much to work with. In the original crowd flow papers with years of\nhalf-hourly data, the trend branch captures genuine long-term shifts in\npopulation movement. Here, it's essentially learning a constant.
\nDespite losing on aggregate, ST-ResNet has one clear advantage: it's better at\npredicting seasonal transitions.
\nThe months where crime shifts gears (the spring uptick in September/October and\nthe February dip) ST-ResNet handles more gracefully than ConvLSTM. The period\nbranch, sparse as its data is, does capture enough of the annual rhythm to\nanticipate these transitions a bit earlier.
\nConvLSTM tends to lag these transitions by about a month. It needs to "see" the\nuptick starting before it predicts continuation. ST-ResNet, by explicitly\nlooking at last year's same month, can anticipate the shift before it fully\nmaterialises in the recent sequence.
\nFor an operational forecasting tool, that one-month lead time on seasonal\ntransitions could be valuable. But in our test set metrics, it's a small\nadvantage that doesn't overcome ST-ResNet's overall weaker performance on\nmonth-to-month dynamics.
\n| Metric | \nHistorical Avg | \nConvLSTM | \nST-ResNet | \n
|---|---|---|---|
| Overall MAE | \n0.39 | \n0.35 | \n0.36 | \n
| Theft MAE | \n1.28 | \n1.14 | \n1.18 | \n
| Training time (CPU) | \nN/A | \n~40 min | \n~35 min | \n
| Parameters | \n0 | \n~200k | \n~180k | \n
| Seasonal transitions | \nPoor | \nLagging | \nBetter | \n
| Spatial dynamics | \nNone | \nGood | \nGood | \n
ConvLSTM is the better model for this specific dataset. Not by a lot. We're\ntalking about small differences on already-small error values. But consistently\nbetter on the main crime types that have enough signal to matter.
\nNeither model is a revelation. A 7–10% improvement over "just use the historical\naverage" is real but modest. Deep learning's strengths (learning complex\nnonlinear dynamics from huge datasets) are somewhat wasted on 48 monthly\ntimesteps over a relatively low-crime city.
\nIf I had daily data instead of monthly, or ten years instead of four, I'd expect\nST-ResNet to close the gap or pull ahead. Its architecture is fundamentally\nsound. The temporal decomposition is a genuinely good idea. It's just starved of\nthe data it needs to shine.
\nBoth models meaningfully beat the baselines. Both learn spatial patterns that\nsimple averages can't capture. And both are honest about the sparse crime types:\nthey predict near-zero and move on, which is the right call.
\nNext up: we'll take these predictions and build something you can actually look\nat. A 3D interactive dashboard where you can watch crime patterns evolve across\nAuckland over time. The modelling was the hard bit. Making it visual is the fun\nbit.
\n","date_published":"Thu, 23 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/three-ways-to-look-at-time.png"},{"id":"https://jonno.nz/posts/what-an-hour-of-your-attention-is-worth/","url":"https://jonno.nz/posts/what-an-hour-of-your-attention-is-worth/","title":"What an hour of your attention is worth","content_html":"I stood up a working social network for eight mates last weekend. Profile pages,\na shared feed, a photo wall, a jukebox bolted onto a spare domain. It took me a\nSaturday, about forty bucks in Claude credits, and exactly zero\nproduct-market-fit meetings.
\nThe same weekend, Meta earned about six bucks off me. Google made ten. LinkedIn,\nYouTube, TikTok, X — all quietly billing in the background, none of them sending\na receipt. If you add them all up for the average American, the annual total is\nnorth of $1,000. You just never see it, because no money changes hands and no\ninvoice arrives.
\nThe clever thing about "free" on the internet isn't that the trade doesn't\nexist. It's that it's been designed so you can't see it. No money moves. No\ninvoice lands. No app shows you the meter ticking as you scroll. The exchange is\nreal — your attention and your data in, Instagram and Google and LinkedIn out —\nbut by the time the numbers get tallied, they live in a quarterly earnings\nreport you'll never read. So the trade feels weightless.
\nIt isn't. You just can't see the price tag.
\nThe strange thing is the price tag has been public the whole time. Every\nplatform listed on a stock exchange tells you, four times a year, exactly what\nyou're worth to them. You've just never been shown how to read it — and until\nrecently, the only practical alternative to reading it was "live in a cabin."\nThat part has changed, and it's the part almost nobody is talking about.
\nThe invisibility isn't an accident either. If Meta had to send you a cheque\nevery month for the money they made off you, you'd treat the relationship very\ndifferently. You'd notice when the amount went up. You'd notice that the\nteenager version of the payment looks nothing like the adult version. You'd\nwonder why the Auckland cheque was ten times the Jakarta one for the exact same\nhour of scrolling. The whole edifice of "free" rests on keeping the accounting\none-sided — they measure you in basis points to three decimal places, you\nexperience the trade as a vague sense of having lost your afternoon.
\nThe number you want is called ARPU — average revenue per user. Every public\nplatform reports it, because investors demand it. The maths is blunt: take the\ncompany's annual revenue, divide by monthly active users. What comes out is what\nthe platform earns off the average human who shows up, per year.
\nFor Meta last year the global figure was about $52 per user. For YouTube's\nad-supported side, around $24. For\nLinkedIn it's $15 averaged across all 1.2B members,\nbut much higher once you strip out the dormant accounts.
\nThese aren't guesses from a watchdog group. They're from the companies\nthemselves, in the part of the earnings release where the whole purpose is to\nconvince shareholders each user is worth more than last quarter. The incentive\nis to talk the number up, not down.
\nWhatever ARPU says, the reality on the ground probably isn't lower. If anything,\nit's a floor.
\nRough figures, from the companies' own filings:
\nThe geographic skew is the part most people miss. Meta's figure in the US is\nroughly ten times what it is in Asia-Pacific. Europe sits in the middle at about\n$92. Same product, same features, same algorithm — different rate card, because\nad buyers pay more to reach wealthier audiences. You are literally worth more in\nAuckland than you are in Jakarta, and your feed is tuned accordingly.
\n![\"Meta](\"https://jonno.nz/img/posts/arpu-meta-by-region.svg\")
The same skew shows up across every ad-funded platform. The US rate card is the\none the rest of the world gets compared to:
\n![\"US](\"https://jonno.nz/img/posts/arpu-us-vs-global.svg\")
Marketplaces don't fit ARPU cleanly, but the extraction is still there if you\nlook for it. Uber and Lyft take around 20% of each fare. Airbnb combines host\nand guest fees for about 14–16%. DoorDash and Uber Eats take closer to 25%.\nShopify's card take is 2.9% plus 30 cents per transaction. Different mechanism,\nsame game — a percentage of every transaction, quietly skimmed, never itemised.
\nARPU is annual. Attention isn't spent in years though — it's spent in hours, in\nthe little windows between other things. So the honest conversion is to divide.
\nThe average US Meta user burns about 200 hours a year across Facebook and\nInstagram. $320 ÷ 200 = roughly $1.60 per hour of your attention. YouTube works\nout to about $0.27/hour. TikTok $0.22. Snapchat cheaper still. Do the same sum\non global averages and Meta drops to around 26 cents an hour, YouTube to 8.
\nThose rates are only what the platform earns this year, mind. They aren't what\nyour data is ultimately worth. Everything you click and hover and pause on\nfeeds ad targeting across the wider web, plus — now — AI training corpora. ARPU\nis the rent. The equity is bigger, and the equity compounds.
\nThe AI-training bit is genuinely new and worth pausing on. For fifteen years the\ndata you generated on these platforms powered one thing: better ad targeting on\nthose same platforms. It was a closed loop. You scrolled, they learned, they\nsold the targeting back to advertisers, the advertisers bought your attention\nagain. Bounded. Weird, but bounded.
\nThat loop isn't bounded anymore. Your posts and comments and DMs are now\ntraining data for models that will be sold, resold, and embedded into every\npiece of software you touch for the next decade. The $320 Meta earned off you in\nthe US last year is a rounding error next to what the underlying corpus is worth\nto the next generation of AI products. ARPU doesn't capture any of that. It's\nliterally last quarter's ad rent, with none of the capital gains on the asset.
\nEven the rent, laid out per hour, makes one thing obvious: you can see exactly\nwhy every platform is obsessed with "time spent" as a north-star metric. If one\nextra hour a week on Facebook is worth ~$83 a year per US user, multiplied\nacross three billion users, the maths for why the feed never stops scrolling is\nnot mysterious. The feed is a meter. Keeping it running is the business. Every\n"new feature" that shows up in your settings — reels, shorts, a nudge to open\nthe app on your commute — is a hand on that meter.
\nOnce you see it that way, a lot of product decisions stop looking like product\ndecisions.
\nThe point of making the numbers this concrete is that you can plug in your own\nusage and see what you personally throw into the machine each year. Drag the\nsliders for how much time goes into each platform and watch the ledger tally up.\nRates are global averages.
\nARPU is rent, not equity. What a platform earns this year isn't what the underlying data is worth across the wider web and AI training corpora.
Averages hide heavy users. Freemium smears free and paying users into one figure. If you're all-in, you're worth more than average.
Multi-product companies cheat the top line. Google's per-user number isn't all Search — it's Search plus Android plus YouTube plus Cloud.
The rates come from the earnings-report maths above — global ARPU divided by\naverage annual hours on the platform.
\nOnce the number has somewhere to sit, it's much harder to ignore.
\nMost people look at a total over $1,000/yr and go quiet for a second. Not\nbecause any one platform is egregious — on a per-hour basis they really aren't —\nbut because the aggregate is real, and it's been invisible until now. That's the\nfirst useful thing the exercise does. It makes a choice possible.
\nThe obvious next move is to look at alternatives. Signal instead of WhatsApp.\nKagi or Brave Search instead of Google. Paid Spotify instead of ad-supported\nSpotify. Bluesky or Mastodon instead of X. Fastmail instead of Gmail. None are\nperfect, and some cost actual money — but once you can price what you're\ncurrently "not paying", the paid alternative often looks less expensive than it\ndid five minutes ago. Fastmail at\n$5/month stops being a luxury when the honest comparison is "$60/yr vs being the\nproduct for an ad network that paid $500 for me last year."
\nThat's the defensive move. It's the one everyone talks about, every time one of\nthese pieces gets written. You switch to the more honest vendor, you feel\nslightly better, and the fundamental shape of the market doesn't move.
\nThe more interesting move is what's happened on the build side, and it's the\npart almost nobody has internalised yet.
\nStanding up a social app used to take a small team months. You needed a backend\nengineer, a frontend engineer, a designer, probably a DevOps person, and a spare\nthree months. That was the real moat — not the network effects, not the\nalgorithm, but the sheer human-hours required to put a working thing on the\ninternet. That's why the only viable answer for twenty years was to build\nsomething big enough to run ads against. Small social didn't exist because small\nsocial couldn't pay the salaries.
\nWith Claude Code, Cursor, v0, and Lovable, that equation has quietly inverted. A\nprofile page, a shared feed, a wall for photos, maybe a jukebox, a chat wall — a\nMySpace-sized thing for you and a dozen friends, on a domain you own, with none\nof it feeding anyone's ad platform — is a weekend. I know because I just did it.\nNot as some Silicon Valley startup trying to replace Facebook. As a Saturday\nproject for eight mates, on a domain that cost twelve bucks, running on a box\nthat costs ten a month.
\nThe bill of materials is embarrassingly short. A boring Postgres. A boring\nNext.js app. Auth via magic link. Storage for photos. An LLM for the fiddly bits\nnobody wants to write from scratch. All of it plumbed together in an afternoon\nof prompting, an evening of cleanup, and a Sunday of adding the jukebox because\nmy mate Hamish wouldn't stop asking.
\nIt is not good software. It is good enough software for eight humans who know\neach other.
\nThat qualifier is the whole thing. Facebook has to be good software at planet\nscale because Facebook is selling ad impressions at planet scale. A group of\neight doesn't need p99 latency and a content moderation policy. A group of eight\nneeds a place to put photos from the weekend where the photos don't end up\ntraining someone's image model in twelve months' time. Those are very different\nengineering problems, and the second one is much, much easier than the first.
\nA lot of things genuinely don't work on the weekend version. There's no\nrecommendation algorithm. There's no real search. The feed is\nreverse-chronological and that's it. When someone posts something at 3am nobody\nsees it until the morning. There's no cleverness about which photos get surfaced\nor which memories get resurrected. If you go on holiday for two weeks, you come\nback to a feed that's exactly what your eight mates posted, in the order they\nposted it.
\nThat sounds like a limitation until you notice the thing it is not doing is\noptimising for your engagement. Reverse-chronological across eight friends is\nnot a meter. It's a wall. You check it, you see what's there, you leave. There's\nno reason for the software to try to keep you around because there's nobody\npaying the software to keep you around. That inversion — from meter to wall — is\nthe entire point.
\nThe thing that would have been a VC round in 2015 is now a side quest you finish\nbefore the roast is in the oven. The tools genuinely got that much better in the\nlast eighteen months. We just haven't updated our intuitions yet about what that\nmeans.
\nWhat it means, specifically, is that the ad-supported social network is no\nlonger the only technically viable answer. For twenty years it was. That was the\nconstraint the whole "free web" was built around. The constraint is gone, and\nnobody has sent the memo.
\nThe cheapest social network in 2026 is the one you and six mates build on a\nSaturday afternoon. It doesn't scale. It doesn't need to. It costs less than a\nmonth of Netflix, produces no ad revenue for anyone, and feeds no one's training\nset. You own the domain. You own the data. You own the product decisions — which\nin practice means there are no product decisions, because nobody is trying to\nsqueeze another hour out of anyone's week.
\nNone of this replaces the platforms, to be clear. You still need Gmail for the\nrecruiter, LinkedIn for the job hunt, YouTube for the tutorial, WhatsApp for the\ngroup chat your family refuses to leave. The ad-supported internet isn't going\nanywhere and I'm not pretending it is. What's changed is that it's no longer the\nonly game in town. For the circle of people you actually care about — the eight\nmates, the cousins, the old uni flat — you don't have to hand them over to the\nad machine anymore. You can build them a room of their own, and the tools to\nbuild that room have become trivial in a way we haven't fully absorbed yet.
\nThe meter's been running your whole life. You just got the tools to turn it off.
\n","date_published":"Tue, 21 Apr 2026 12:00:00 GMT","image":"https://jonno.nz/og/what-an-hour-of-your-attention-is-worth.png"},{"id":"https://jonno.nz/posts/teaching-a-neural-network-to-watch-crime-like-video/","url":"https://jonno.nz/posts/teaching-a-neural-network-to-watch-crime-like-video/","title":"Teaching a Neural Network to Watch Crime Like Video","content_html":"ConvLSTM was invented to predict rainstorms.
\nSpecifically,\nShi et al. at the Hong Kong Observatory\nneeded to forecast radar echo maps: 2D grids of rainfall intensity that evolve\nover time. They had sequences of spatial images and wanted to predict the next\nframes. Sound familiar?
\nThat's exactly what we built in Part 3. Crime on a 500m grid, one frame per\nmonth, six channels for crime types. The Auckland crime tensor is structurally\nidentical to a weather radar sequence. Same dimensionality, same prediction\ntask, just a very different domain.
\nStandard LSTM networks are fantastic at learning sequences. They're the backbone\nof a lot of time-series forecasting. But they have a fundamental problem with\nspatial data: they need flat vectors as input.
\nTo feed our 77×59 grid into a regular LSTM, we'd have to flatten it into a\nvector of 4,543 values per crime type. That's 27,258 values per timestep across\nall six channels. The network would process this as a sequence of big flat\nvectors, with no concept that cell (10, 5) is next to cell (10, 6).
\nAll the spatial structure (the fact that crime clusters, that hotspots have\nneighbourhoods, that the CBD is a contiguous area) gets thrown away. The model\nwould have to rediscover spatial relationships from scratch, purely from\ncorrelations in the flattened vector. With only 36 training months, that's not\nhappening.
\nConvLSTM's insight is elegant. Take the standard LSTM equations (the input gate,\nforget gate, output gate, cell state update) and replace every matrix\nmultiplication with a convolution operation.
\nIn a regular LSTM:
\ninput_gate = sigmoid(W_xi * x_t + W_hi * h_{t-1} + b_i)\nIn ConvLSTM:
\ninput_gate = sigmoid(W_xi ∗ X_t + W_hi ∗ H_{t-1} + b_i)\nThat ∗ is a convolution instead of a matrix multiply. X_t is the full 2D\ngrid at time t, and H_{t-1} is the previous hidden state, also a 2D grid.\nThe convolution kernel slides across the spatial dimensions, so each cell's gate\nvalues depend on its local neighbourhood.
This means the network naturally learns that a spike in cell (10, 5) might\naffect predictions for cell (10, 6). Spatial proximity is baked into the\narchitecture. It doesn't need to learn it from data.
\nThe kernel size controls how much spatial context each cell sees. A 3×3 kernel\nmeans each cell looks at its immediate 8 neighbours. Stack multiple ConvLSTM\nlayers and the effective receptive field grows. Deeper layers can capture\nrelationships between cells that are several kilometres apart.
\nHere's what I settled on after a fair bit of experimentation (which on CPU means\n"a lot of patient waiting"):
\nInput: (batch, 6, 6, 77, 59), 6 months, 6 crime types, 77×59 grid\n ↓\nConvLSTM2d(in=6, hidden=32, kernel=3×3, padding=1)\n ↓\nBatchNorm2d\n ↓\nConvLSTM2d(in=32, hidden=32, kernel=3×3, padding=1)\n ↓\nBatchNorm2d\n ↓\nConv2d(in=32, out=6, kernel=1×1), project to 6 crime type channels\n ↓\nOutput: (batch, 6, 77, 59), next month prediction\nTwo ConvLSTM layers with 32 hidden channels each. The 3×3 kernel gives each cell\na neighbourhood view, and stacking two layers means the effective receptive\nfield covers about 1–1.5 km. Enough to capture the spatial extent of most crime\nhotspots.
\nWhy only 32 hidden channels? This is where the CPU constraint actually helps. A\nbigger model would be tempting with a GPU, but on a Ryzen 5 we need to keep it\ntight. 32 channels gives us about 200k trainable parameters: small enough to\ntrain in under an hour, large enough to learn meaningful spatial-temporal\npatterns.
\nThe 1×1 convolution at the end is a channel projection. It maps the 32 learned\nfeatures back to 6 crime type predictions.
\nThe lookback window is six months. The model sees January through June and\npredicts July. Then February through July to predict August. And so on.
\nSix months captures one half of the seasonal cycle, which turned out to be the\nsweet spot. Shorter sequences (3 months) missed seasonal context. Longer\nsequences (12 months) didn't improve results, likely because the model doesn't\nhave enough data to learn year-long dependencies with only 36 training months\ntotal.
\nThe training set gives us 30 sequences (months 1–6 predict 7, months 2–7 predict\n8, all the way to months 30–35 predict 36). That's not a lot. Every sequence\ncounts.
\noptimiser = Adam(lr=1e-4)\nloss = MSE # on log1p-transformed values\nbatch_size = 4 # small because sequences are large\nepochs = 150 with early stopping (patience=15)\nThe log1p transformation from Part 3 is critical here. Raw crime counts range\nfrom 0 to 50+. After log1p, the range compresses to 0–4. Without this, the\nloss function would be dominated by the handful of high-count CBD cells, and the\nmodel would essentially ignore the rest of the grid.
Training on CPU takes about 40 minutes per run. Not fast, but manageable. I\ncould typically fit in 3–4 experimental runs per evening, which meant progress\nwas slow but steady. Each run I'd tweak one thing (kernel size, hidden channels,\nlearning rate) and compare validation MAE.
\nEarly stopping triggers around epoch 80–100 in most runs. The model converges\nrelatively quickly, which makes sense given the small dataset and architecture.
\nSo how does ConvLSTM stack up against the baselines from Part 5?
\n| Crime Type | \nHist. Avg MAE | \nConvLSTM MAE | \nImprovement | \n
|---|---|---|---|
| Theft | \n1.28 | \n1.14 | \n10.9% | \n
| Burglary | \n0.35 | \n0.32 | \n8.6% | \n
| Assault | \n0.20 | \n0.19 | \n5.0% | \n
| Robbery | \n0.04 | \n0.04 | \n2.5% | \n
| Sexual | \n0.03 | \n0.03 | \n~0% | \n
| Harm | \n0.01 | \n0.01 | \n~0% | \n
| All types | \n0.39 | \n0.35 | \n10.3% | \n
A 10% improvement on the aggregate MAE. Not earth-shattering, but real.
\nTheft gets the biggest lift because there's the most signal to work with. The\nmodel genuinely learns spatial dynamics that the historical average can't\ncapture. When a cluster of cells in South Auckland trends upward over several\nmonths, ConvLSTM picks up on that momentum and adjusts its predictions\naccordingly.
\nBurglary sees a decent improvement too, likely driven by the spatial correlation\nwith theft that we spotted in the EDA.
\nFor the sparse crime types (robbery, sexual offences, harm) ConvLSTM basically\nlearns to predict near-zero, same as the baseline. There simply isn't enough\nsignal at 500m monthly resolution for these types. The model is honest about\nwhat it doesn't know, which I actually respect.
\nThe improvement isn't uniform across the grid. ConvLSTM does best in the\ntransition zones: cells on the edges of established hotspots where crime counts\nfluctuate month to month. It learns that these boundary cells tend to follow the\ntrend of their neighbours, which is exactly the kind of spatial-temporal pattern\nit was designed to capture.
\nIn the stable hotspot cores (the CBD, Manukau) the model performs about the same\nas the baseline. Those cells are consistently high, and the historical average\nalready captures that well.
\nWhere it properly struggles is with sudden spikes in normally quiet areas. A\ncell that's been near-zero for months and then gets 5 thefts in one month: the\nmodel doesn't see that coming. Neither does any other model, to be fair. Those\nevents are closer to random noise than learnable signal.
\nA 10% MAE improvement is meaningful but modest.\nRecent ConvLSTM crime prediction papers\nreport larger gains, but they typically work with much more data: years of daily\nrecords across cities with higher crime density. Our setup is tougher. Monthly\nresolution limits temporal signal, Auckland is relatively low-crime by global\nstandards, and we only have four years.
\nThe model is also running on CPU with a deliberately small architecture. A\nbigger model on a GPU might squeeze out more performance. But the point of this\nproject was always to see how far you can push it with modest resources, and a\n10% beat over simple baselines feels like a real result.
\nThe question now is whether ST-ResNet's different approach to temporal modelling\ncan do better. ConvLSTM processes time as one continuous sequence. ST-ResNet\nbreaks it into three separate temporal scales: closeness, period, and trend.\nWith a seasonal dataset like crime, that decomposition might be exactly what's\nneeded.
\n","date_published":"Thu, 16 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/teaching-a-neural-network-to-watch-crime-like-video.png"},{"id":"https://jonno.nz/posts/open-source-agent-that-teaches-claude-code-your-architecture/","url":"https://jonno.nz/posts/open-source-agent-that-teaches-claude-code-your-architecture/","title":"Open-Source Agent That Teaches Claude Code Your Architecture","content_html":"AI has made building software cheap. A solo founder with Claude Code or Cursor\ncan ship an MVP in a weekend that would've taken a small team a month two years\nago. I've watched this happen across the NZ startup scene. Ideas that used to\ndie in the "can we afford to build it" phase now get built over a long weekend.
\nThis is mostly great. Velocity is what startups need. Cost of testing an idea is\nnow close to zero, and the business prioritises speed.
\nThe catch shows up when the idea works.
\nAI builds for right now. It optimises for the current prompt, the current\nfile, the current feature. It doesn't think about what happens when your billing\nservice needs to handle 10x the volume, or when your email notifications need to\nmove from inline calls to a queue. It doesn't plan for the evolutionary pressure\nyour system will face once it has users.
\nThat's the gap I've been thinking about, and it's what led me to build\ndomain-agents.
\nI want to be fair to the current generation of AI coding assistants. They're not\nstupid about finding code.
\nClaude Code runs an agentic search loop (grep, glob, file reads) iterating\nthrough your codebase to find what's relevant. Boris Cherny (who created Claude\nCode) has said they tried\nRAG with a local vector database early on and dropped it because agentic search\noutperformed it. Cursor takes a different approach: it\nchunks your codebase, generates embeddings,\nand stores them for semantic search so you can find code by concept rather than\nkeyword. Copilot combines semantic indexing with LSP-powered reference tracing\nfrom VS Code.
\nThe search works. If you ask Claude Code to find your billing service, it'll\nfind it. Ask Cursor for authentication logic and the embeddings will surface it\neven if the code never uses the word "authentication."
\nNone of them understand the architecture those files live in.
\nAll the information needed to understand domain relationships sits in the code:\nimport graphs, interface signatures, dependency patterns. These tools don't\nextract or structure it that way. They find files one at a time. They don't map\nout that your billing service depends on the email service, that\nBillingService is consumed by two other domains, or that changing its\ninterface is a cross-domain event. The information is in the codebase. Nobody's\npulling it together.
And every session starts from zero. The AI learned your architecture yesterday\nand forgot it today.
\nMy thesis: cheap AI-built MVPs plus expensive scaling problems point toward\nevolutionary architecture with domain-based boundaries.
\nThe idea isn't new. The reason it matters now is.
\nIn an evolutionary architecture, you focus on clean interfaces between business\ndomains. Your email service exposes a contract like\nsendEmail(to, subject, body), and the rest of the system calls that interface.\nBehind the interface, the implementation evolves through stages as your scaling\nneeds change:
graph LR\n A["Inline\\n(direct call)"] --> B["Async\\n(fire & forget)"]\n B --> C["Queued\\n(BullMQ/SQS)"]\n C --> D["Separate Service"]\n D --> E["Distributed"]\nDay one, sendEmail is a function that calls Resend directly. Inline,\nsynchronous, dead simple. When traffic picks up, you drop the await and let it\nrun in the background. Later, you introduce BullMQ or SQS. Eventually it becomes\nits own service. The interface stays put. Only the implementation behind it\nchanges.
This is the kind of evolution AI coding assistants are terrible at planning for.\nThey'll inline that email call because it works right now. They have no\nconcept of where this domain sits on its scaling trajectory.
\ndomain-agents is a CLI tool that\nruns static analysis on TypeScript codebases, discovers business domains, and\ngenerates AI agent context files for Claude Code and Cursor.
\ndomain-agents discover . # Analyse codebase → proposal.json\ndomain-agents init . # Generate agents/*.md + AGENTS.md\ndomain-agents hooks claude # Wire into Claude Code (rules + MCP server)\ndomain-agents hooks cursor # Wire into Cursor (.mdc rules)\nAfter setup, opening src/billing/invoice.ts in Claude Code loads the billing\ndomain agent into context. The AI now knows: billing depends on email (coupling\nscore 0.23), exposes BillingService consumed by 2 other domains, sits at the\n"inline" scaling stage with a path toward async queuing, and has 3 tracked tech\ndebt items.
It plans work accordingly. The context was loaded before the first prompt, no\nsearch required.
\nThe discovery engine runs 5 analysis passes because no single signal identifies\nbusiness domains on its own.
\nDirectory structure works for greenfield projects (src/auth/, src/billing/)\nbut fails for legacy MVC apps. Import graphs capture coupling but not business\nintent. Package dependencies hint at external integrations but miss internal\ndomains.
graph TD\n S["Structure Analysis"] --> O["Signal Orchestrator"]\n I["Import Graph\\n(TS Compiler API)"] --> O\n N["Naming Patterns"] --> O\n D["Dependency Mapping\\n(npm → domain hints)"] --> O\n IF["Interface Detection"] --> O\n O --> M["Merge Pipeline"]\n M --> R["Domain Proposal"]\nStructure detects whether the codebase is feature-organised,\nlayer-organised, mixed, or flat. Import graph uses the TypeScript Compiler\nAPI to parse each .ts file, resolve imports, and build a directed edge graph.\nType-only imports get weighted at 0.3 because they're a weaker coupling signal\nthan value imports. Naming patterns extract domain prefixes:\nauth.controller.ts → "auth". Dependency mapping maps npm packages to\ndomain hints (stripe → billing, @sendgrid/mail → email). Interface\ndetection identifies files imported across domain boundaries and calculates\ncoupling scores between domain pairs.
Each pass produces weighted signals. The orchestrator combines them with\nconfidence scoring: average signal strength plus a bonus for signal count,\ncapped at 0.99. Layer-organised codebases get an 0.85 multiplier because they're\nharder to discover.
\nFeature-organised codebases are easy. The directory structure is the domain.\nBut most real codebases look like this:
\nsrc/\n controllers/\n auth.controller.ts\n billing.controller.ts\n services/\n auth.service.ts\n billing.service.ts\n models/\n invoice.model.ts\n user.model.ts\nHere auth.controller.ts, auth.service.ts, and auth.routes.ts all belong to\nthe "auth" domain despite living in three different directories. domain-agents\nuses naming pattern extraction cross-referenced with import graph cohesion to\ncluster these. The auth.* files form a tight import cluster, which confirms\nthe naming signal.
Raw signals produce too many small, overlapping clusters. The orchestrator runs\na multi-phase normalisation pipeline.
\nPlurals merge: journals + journal → whichever has more files. Compound names\nconsolidate: bank-balance + bank-statement + bank-transaction →\nbank-accounts (the largest cluster). Small clusters merge into their strongest\nimport target, but only if they have a dominant dependency: more than 40% of\nimports from one target, and that target is at least 2x larger. This prevents\ncascading, where A merges into B, B gets bigger and attracts C, C pulls in D.
Files that import from 3+ domains get moved to "unassigned." These are coupling\nhotspots: middleware, orchestrators, shared handlers. Assigning them to one\ndomain would mislead the AI, so the tool surfaces them for a human decision.\nThat's the right call for architectural boundaries.
\nThe E2E test suite validates the complete pipeline against 3 fixture codebases\n(feature-organised, layer-organised, mixed). Current benchmark: 100% activation\naccuracy across all 3 patterns and all 3 activation levels (domain assignment,\nglob matching, MCP lookup).
\nThe integration into Claude Code and Cursor uses glob-based rule activation, the\nnative mechanism both tools already support.
\nEach domain gets a rule file with glob patterns in the frontmatter:
\n---\ndescription: billing domain\nglobs:\n - src/billing/**\n - **/billing.*\n - **/billing-*\n---\nWhen Claude Code opens a file matching those globs, the domain context loads. No\nMCP call, no background process, zero runtime overhead.
\nAn MCP server complements the rules with 4\non-demand tools: domain_lookup(file), domain_context(name),\ndomain_files(name), and list_domains(). A SessionStart hook prints a domain\nsummary at the start of every Claude Code session, so the AI has system-level\nawareness from the first prompt.
This is the bit I'm most keen on long-term.
\nAt Vend and Xero, teams owned domains. The billing team owned billing, the\nintegrations team owned integrations. Ownership meant knowing the interfaces,\nthe coupling points, the tech debt, and where things were headed. That knowledge\nlived in people's heads and got passed on through code reviews, architecture\nchats, and tribal memory.
\nDomain-specific AI agents formalise that same ownership model. An email agent\nloads the email domain's interface contract, its coupling to other domains, its\ncurrent scaling stage, and its tracked tech debt. A billing agent carries the\nsame for billing. They work within their boundaries and flag when a change\ncrosses a domain line.
\nYou don't need this from day one. Early on, one agent covers multiple areas. As\nthe product grows, agents split along the same lines engineering teams split: by\nbusiness domain. The operator (that's you) resolves conflicts where agents\ndisagree, the same way an engineering manager resolves cross-team dependencies.
\nThe analogy is rough, but it captures how AI-assisted development scales past a\nsingle person staring at a single context window.
\n","date_published":"Wed, 15 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/open-source-agent-that-teaches-claude-code-your-architecture.png"},{"id":"https://jonno.nz/posts/openhealth-chat-with-apple-health-data/","url":"https://jonno.nz/posts/openhealth-chat-with-apple-health-data/","title":"OpenHealth – Chat with Apple Health Data, Anywhere","content_html":"For years I've worn an Apple Watch and let my iPhone quietly hoover up my\nresting heart rate, HRV, sleep stages, every workout, every nutrition log.\nMillions of data points. And for most of that time, when I wanted to actually\nask something about my training — "am I cooked this week?", "has my recovery\ngotten worse since Christmas?" — I'd open ChatGPT and get an answer that was\nbasically vibes, because it couldn't see any of the data.
\nSo I built openhealth. It turns your\nApple Health export into seven short markdown files any LLM can read. Drop the\nzip in your browser at\nopenhealth-axd.pages.dev, run the CLI, or\nbeam the zip straight from your iPhone over WebRTC. Paste the output into Claude\nor ChatGPT and start asking the questions you actually wanted to ask.
\n![\"openhealth's](\"https://jonno.nz/img/posts/openhealth/hero.png\")
In January, Anthropic\nshipped an Apple Health connector\nfor Claude. OpenAI has one in ChatGPT. Both are US-only — if you're in New\nZealand like me, or the UK, EU, or Switzerland,\nthey're not available. That's\na lot of people locked out of the most natural way to use this data.
\nAnd even if you are in the US, you're letting Anthropic or OpenAI decide what\nthe model reads, how it's framed, and what tier unlocks it. I wanted control\nover the whole pipeline — including which LLM I feed it into.
\nopenhealth ships three ways.
\nA static web app. Drop export.zip, wait five seconds, download seven\nfiles. The browser does the parse. There's no upload endpoint because there's no\nserver — the Cloudflare Pages site is static HTML plus a tiny Web Worker. Open\nDevTools, watch the Network panel, nothing goes out.
A Bun-compiled CLI. openhealth ~/export.zip -o ./output gets you seven\nmarkdown files. --bundle concatenates them into one. --clipboard pushes that\nbundle straight to your system clipboard so you can paste it into any chat\nwindow. Zero deps beyond saxes for XML and fflate for unzip — even the\nargument parsing is node:util parseArgs, not Commander. One binary, put it\nwherever.
A phone-to-desktop handoff over WebRTC. The desktop site renders a QR code.\nPoint your iPhone camera at it, Safari opens a tiny receiver page, pick the zip,\nand it streams directly to your desktop browser over a DataChannel. The only\nbackend in the whole stack is a ~100-line Cloudflare Worker that relays the\nWebRTC handshake — it never sees a byte of your health data.
\n![\"Getting](\"https://jonno.nz/img/posts/openhealth/walkthrough.png\")
Apple's export.xml is\nproperly huge.\nA long-term Watch user can easily have a 500MB–4GB file with millions of rows.\nMost XML parsers build a tree in memory, which OOMs before they finish.
openhealth uses saxes — a streaming SAX\nparser in pure TypeScript. It's isomorphic, so the same parser runs in Bun,\nNode, and the browser. I tested it against a synthetic 169MB / 1 million-record\nexport and it finished in about 5 seconds in Chrome, with the main-thread heap\nstaying around 5MB because the parse runs in a Web Worker.
\nThe rest of the core is a small pipeline: stream XML, accumulate\nper-record-type, roll up into weekly and monthly summaries, run each through a\nwriter that produces one markdown file. Every writer is snapshot-tested against\nbyte-for-byte expected output. 85 tests, TDD throughout.
\nEach one is deliberately small and shaped to be LLM-readable:
\nhealth_profile.md — baselines, data sources, long-term averagesweekly_summary.md — current week plus a 4-week rolling comparison with\nweek-over-week deltasworkouts.md — detailed log for the last 4 weeks: HR, duration, distance,\nenergybody_composition.md — weight trend, recent readings, nutrition averagessleep_recovery.md — nightly stages, 8-week averages, HRV, resting HR, SpO2\ntrendscardio_fitness.md — running log, HR-zone distribution, walking-speed trendsprompt.md — a ready-to-paste system prompt that frames the other six as\ncoaching inputDrop one file or all seven, depending on which chat model you're using.
\nFeeding real data to an LLM is a different experience from answering its\nquestions. When Claude can see that my resting HR has crept up 4bpm over the\nlast fortnight while my HRV has dropped and my training load stayed the same, it\ngives a real answer — "you're likely undercooked on recovery this week, here's\nwhat I'd change" — rather than a generic reminder to drink water.
\nIt's especially good if you've got multiple devices in the mix. I've got data\nfrom Apple Watch, the iPhone step counter, a Withings scale, and MyFitnessPal.\nThe parser picks the highest-trust source per metric — Apple Watch wins over\niPhone for steps, Watch sleep beats AutoSleep which beats Withings,\nduplicate-weight entries on the same day get deduped. You feed in one zip and\nget one coherent picture.
\nAsk it about your recovery, your training load, what you might be doing wrong,\nhow your sleep correlates with your long runs. It'll tell you — and it'll be\nright more often than not.
\nopenhealth itself never uploads your data. The web app parses in your browser\ntab. The CLI runs locally. The WebRTC handoff stays peer-to-peer — the\nCloudflare Worker that relays the handshake never sees a byte of the file. Clone\nthe repo, diff the build output, and confirm it yourself.
\nWhen you paste the seven files into ChatGPT or Claude, they see the data.\nThat's the trade most people will take for convenience, and it's fine. But if\nyou don't want to make that trade, you don't have to — run the CLI and pipe the\nbundle into a local model:
\nopenhealth ~/export.zip --bundle -o ./out\nollama run llama3 < ./out/openhealth.md\nOllama, llama.cpp, LM Studio, whatever you run. Your health data never leaves\nyour laptop. The output is just markdown — it doesn't care what reads it.
\nThat's why the shape is seven files and not an API. You pick what sees them.
\nI'm not a doctor. Neither is the model. Use this for thinking out loud about\nyour own training, not diagnosing anything.
\nMIT, source at\ngithub.com/jonnonz1/openhealth. Web\napp at openhealth-axd.pages.dev. If you've\nbeen sitting on a 200MB export.zip with nothing that'll open it, have a go.
The moment this project went from "fun weekend hack" to something I actually use\nevery day was when I got the MCP server working. Claude Code on my laptop sends\na prompt to the orchestrator sitting under my desk, which boots a VM, runs\nClaude Code inside it with full permissions, and streams the results back.\nClaude delegating work to Claude.
\nIt's a weird feeling watching it happen. You're in a conversation with Claude,\nit decides a task needs isolation, calls the MCP tool, and a few seconds later\nyou can see a fresh VM spinning up in the dashboard. Like having an intern who\ncan clone themselves.
\nPart 1\ncovered why I built this.\nPart 2 was the\nguts of it — rootfs, networking, the guest agent. This last post is about the\ninterfaces, the streaming pipeline, and what I'd change if this needed to work\nfor more than just me.
\nThe orchestrator exposes an MCP server with\neight tools. The main one is run_task — give it a prompt, optional config\n(RAM, vCPUs, timeout, max turns), and it blocks until the task completes.\nReturns the task ID, status, exit code, result files, cost, and the output\ntruncated to 4000 characters.
Two transport modes. Stdio for when Claude Code runs on the same machine:
\n{\n "mcpServers": {\n "orchestrator": {\n "command": "sudo",\n "args": ["/opt/firecracker/bin/orchestrator", "mcp"]\n }\n }\n}\nAnd Streamable HTTP for network access — Claude Code on any machine on the LAN\ncan use it:
\n{\n "mcpServers": {\n "orchestrator": {\n "type": "http",\n "url": "http://192.168.50.44:8081/mcp"\n }\n }\n}\nThe other tools are for poking around: get_task_status, list_vms,\nexec_in_vm (run a command in a still-running VM), read_vm_file,\ndestroy_vm, list_task_files, and get_task_file. That last one is smart\nabout content types — text files come back as plain text, images come back as\nbase64 MCP image content so Claude can actually see screenshots the VM took.
if isImageMime(mimeType) {\n encoded := base64.StdEncoding.EncodeToString(data)\n return mcplib.NewToolResultImage("Screenshot from task "+taskID, encoded, mimeType), nil\n}\nThis bit is worth telling because it'll save someone else the debugging time.
\nI originally built the MCP server with\nmcp-go v0.45.0 using SSE (Server-Sent\nEvents) transport. Worked great. Then Claude Code updated to expect the newer\nStreamable HTTP transport, and everything fell over.
\nThe failure mode was confusing. Claude Code would try to connect, attempt OAuth\ndiscovery against the /sse endpoint, get a 404 (my server doesn't do OAuth),\nand fail with:
Error: HTTP 404: Invalid OAuth error response: SyntaxError: JSON Parse error: Unable to parse JSON string\nNothing in my code changed. The client just started speaking a different\nprotocol.
\nThe fix was small once I understood it:
\n// Before — SSE transport\nfunc (s *Server) ServeSSE(addr string) error {\n sseServer := server.NewSSEServer(s.mcpServer,\n server.WithBaseURL("http://"+addr),\n )\n return sseServer.Start(addr)\n}\n\n// After — Streamable HTTP transport\nfunc (s *Server) ServeHTTP(addr string) error {\n httpServer := server.NewStreamableHTTPServer(s.mcpServer,\n server.WithEndpointPath("/mcp"),\n server.WithStateLess(true),\n )\n return httpServer.Start(addr)\n}\nBumped mcp-go from v0.45.0 to v0.46.0, swapped the server constructor, changed\nthe endpoint from /sse to /mcp, updated the client config. Done. But\ndiagnosing "OAuth error on a server that doesn't do OAuth" — that bit took a\nwhile.
When Claude Code runs inside a VM, its output needs to get from stdout inside\nthe guest all the way to a browser tab on my laptop. The path:
\nflowchart LR\n A["Claude Code stdout"] --> B["Guest agent\\nvsock frame"]\n B --> C["Host vsock client\\nExecStream"]\n C --> D["Task runner\\nOnEvent callback"]\n D --> E["Stream Hub\\nring buffer + fan-out"]\n E --> F["WebSocket\\nto browser"]\nThe stream hub (internal/stream/hub.go) is a per-task pub/sub system. Each\ntask gets a stream with a 1000-event ring buffer. When a WebSocket client\nconnects, it gets all the buffered history first, then live events as they\narrive.
Fan-out is non-blocking:
\nfor ch := range s.subscribers {\n select {\n case ch <- event:\n default:\n // Subscriber is slow, drop the event\n }\n}\nA slow WebSocket client can't block the task runner. If the browser can't keep\nup, it misses events. In practice this never happens because the bottleneck is\nalways Claude thinking, not the network.
\nThe React frontend is compiled to static files and embedded into the Go binary:
\n//go:embed all:web-dist\nvar webDistEmbed embed.FS\nSingle binary deployment. No nginx, no separate frontend server, no CORS\nheadaches in production. The API server falls through to index.html for\nunknown paths, which gives you SPA client-side routing.
The most interesting page is the task detail view. Claude Code's\n--output-format stream-json spits out one JSON object per line — thinking\nblocks, text responses, tool calls, tool results, cost summaries. The dashboard\nparses these into coloured blocks:
A useWebSocket hook connects when the task is running and disconnects when\nit's done. Green pulsing dot for live streaming. Auto-scroll to the bottom as\nevents arrive. Image files in the results get inline previews pointing at the\nAPI's file download endpoint — so when Claude takes a screenshot inside the VM,\nyou see it immediately.
Dark theme. Orange accents. Obviously.
\nThis runs on one box with no auth. It's a home lab project. But the gap between\n"works for me" and "works for a small team" isn't as big as it looks.
\nPersistence is the most obvious one. The task store is an in-memory Go map.\nOrchestrator restarts? All task history gone. VM metadata already persists to\ndisk and gets recovered on startup — tasks should too. SQLite or bbolt, a few\nhours of work. I just haven't needed it because I don't restart the process very\noften.
\nTask queue with backpressure. Right now tasks fire as goroutines with no\nconcurrency limit. Submit 20 tasks on a 30GB machine where each VM wants 2GB and\nthe last few fail because there's no memory left. A buffered channel or\nsemaphore would fix this. You could get fancier with priority queues — quick\ncode generation tasks ahead of long research tasks — but even a simple\nconcurrency cap would be enough.
\nAuthentication. The REST API and MCP server accept requests from anyone who\ncan reach the port. For a team: API keys at minimum, mTLS if you're serious\nabout it. The MCP spec supports auth flows now — that'd be the right way to do\nit for the MCP endpoint.
\nThe OnEvent callback race. This one's a latent bug. The task runner's\nOnEvent callback is stored on the runner struct, not passed per-task:
s.taskRunner.OnEvent = func(id string, event agent.StreamEvent) {\n taskStream.Publish(event)\n}\ns.taskRunner.Run(context.Background(), t)\nTwo simultaneous tasks overwrite each other's callbacks. It works today because\nMCP tasks block (one at a time) and the API handler sets up the stream before\nthe goroutine runs. But it's the kind of thing that works until it doesn't. Fix\nis trivial — pass the callback into Run() as a parameter.
Graceful shutdown. There's no signal handler. Ctrl-C kills the process,\nrunning VMs become orphans. They keep running as Firecracker processes — the\nrecoverState() function on next startup finds them and starts tracking them\nagain — but their tasks are lost. A proper signal handler would stop accepting\nnew tasks, wait for running ones to finish with a timeout, then tear everything\ndown cleanly.
For real multi-user you'd want result storage on S3 or R2 instead of local\ndisk. A web auth layer. Per-user credential vaults so different people's Claude\ntokens don't mix. Usage tracking and cost attribution.
\nWhat I wouldn't change: the single-binary deployment, vsock for host-guest\ncommunication, ephemeral VMs as the isolation model, the embedded frontend.\nThose are the right calls regardless of scale. The architecture is sound — it's\nthe operational bits around it that need work.
\nMost of these are a weekend each. The project is about 3,200 lines of Go and 860\nof TypeScript. It's not a big codebase. Adding persistence, auth, and a task\nqueue would maybe take it to 4,500 lines. Still fits in your head.
\nFor now, it sits under my desk and boots VMs when I ask it to. Claude delegating\nto Claude, in complete isolation, on hardware I own. That's enough.
\n","date_published":"Mon, 13 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/claude-code-can-now-spawn-copies-of-itself-in-isolated-vms.png"},{"id":"https://jonno.nz/posts/llm-kills-compromised-services-at-3am/","url":"https://jonno.nz/posts/llm-kills-compromised-services-at-3am/","title":"The Future of Security Is an Open-Source Model That Detects and Acts on Threats","content_html":"Anthropic just dropped Project Glasswing\n— a big collaborative cybersecurity initiative with a shiny new model called\nClaude Mythos Preview that can find zero-day vulnerabilities at scale. Twelve\nmajor tech companies involved. $100M in credits. Found a 27-year-old flaw in\nOpenBSD. Impressive stuff.
\nBut let's be real about what's happening here. Anthropic trained a model so\ncapable at breaking into systems that they decided it was too dangerous to\nrelease publicly. So they wrapped the release in a collaborative security\ninitiative. The security work is genuinely valuable. But it's also a smart way\nto keep control of something they know is too powerful to let loose.
\nThe part that actually matters, though, is who benefits. Glasswing is for the\nbig players. The companies with security teams, budgets, and the kind of\ninfrastructure that gets invited to sit at the table with AWS, Microsoft, and\nPalo Alto Networks. What about the rest of us? The startups, the small SaaS\nshops, the indie developers running production systems on a shoestring?
\nThe internet is a\ndark forest.\nThat's not a metaphor anymore — it's becoming the literal reality. Bots,\nscrapers, automated exploit chains, credential stuffing, AI-generated phishing.\nA server goes up and within hours it's being scanned, fingerprinted, and probed\nby systems that don't sleep. Visibility equals vulnerability. And AI is making\nthe attackers faster, cheaper, and more autonomous every month.
\nThe\nISC2 put it plainly\n— both offence and defence now operate at speeds beyond human intervention. The\nthreats aren't people sitting at keyboards anymore. They're autonomous systems\nrunning campaigns end-to-end.
\nSo what do we do about it?
\nWhen I say offensive security, I don't mean red-teaming or penetration testing.\nI mean giving your systems the ability to fight back.
\nPicture an LLM that sits across your centralised logs — network traffic,\ndatabase queries, user interactions, access patterns — and builds an\nunderstanding of what normal looks like for your system over weeks and months.\nNot just pattern matching against known signatures. Actually understanding the\nshape of healthy behaviour.
\nWhen something breaks the pattern, it doesn't just alert. It acts.
\nDisable a compromised account. Kill a service that's behaving strangely. Block a\ndatabase connection that shouldn't exist. Create an incident with full context\nfor a human to review. The response is proportional and immediate — not waiting\nfor someone to check their phone at 3am.
\nThe architecture is pretty straightforward:
\ngraph TD\n A[Application Logs] --> D[Secure Isolated Log Store]\n B[Network Traffic] --> D\n C[Database Queries] --> D\n D --> F[Baseline Health Model]\n E[User Activity] --> D\n F -->|Anomaly Detected| G[LLM Analysis]\n G -->|Analyse & Plan| H{Threat Assessment}\n H -->|Low| I[Alert & Log]\n H -->|Medium| J[Restrict & Escalate]\n H -->|High| K[Disable & Isolate]\n I --> L[Human Review]\n J --> L\n K --> L\nThe key is that the logging and analysis layer has to be isolated and secured\nseparately from the systems it's watching. If an attacker can compromise the\nthing that's watching them, the whole model falls apart.
\nIn practice that means separate infrastructure with its own auth boundary.\nIngestion is write-only — your application services push logs in but can never\nread or modify what's already there. Append-only, immutable. The analysis layer\ngets scoped service accounts that can read logs, fire alerts, and pull specific\nemergency levers through a narrow API. Nothing else. If a compromised service\ntries to reach the log store directly, it hits a wall.
\nNone of this is exotic. Centralised logging, immutable storage, scoped IAM — the\nbuilding blocks exist. The hard part is wiring an LLM into that loop with the\nright constraints. Enough access to act, not enough to make things worse.
\nTraditional security tooling runs on signatures and static rules. Known bad\npatterns, blocklists, threshold alerts. That worked when threats were mostly\nhuman-paced. It doesn't work when you're up against autonomous systems that\nadapt faster than you can write rules.
\nThe alternative is a system that learns what normal looks like for your\nenvironment — not a generic baseline, but the actual shape of healthy behaviour\nin your specific infrastructure. Traffic patterns, query frequencies, access\ntiming, user behaviour. Weeks of observation before it starts making decisions.
\nWhen something breaks the pattern, the response is proportional. A sudden spike\nin unusual API calls might trigger deeper correlation — the system widens its\nsearch, pulls in more signals, lowers its threshold for flagging related\nactivity. Repeated failed auth attempts from new IPs tighten access controls\nautomatically. A database connection that shouldn't exist gets killed.
\nThis isn't a static ruleset you configure once and hope covers everything. It's\na system that develops behavioural intuition from running in your environment,\nresponding to your traffic. The difference matters — static rules are brittle\nagainst novel attacks, while adaptive systems can catch anomalies they've never\nseen before.
\nThe baseline isn't magic. It's watching five things:
\nThree layers of detection stack on top of each other. Layer one is simple\nthresholds — hard caps that trigger immediately. Layer two is statistical\ndeviation — standard deviations from the learned baseline. Layer three is\ncorrelation — looking across multiple signals simultaneously. A spike in rate\nalone might be fine. A spike in rate plus unusual composition plus new source\nIP? That's a pattern.
\nA pure anomaly detector would go nuts during deploys. New code paths, changed\nresponse times, config reloads — all of it looks unusual. Same with cron jobs.\nYour 3am batch job that hits the database hard every night would trigger alerts\nevery night.
\nTolerance patterns solve this. The system learns to recognise you.
\nMark a deploy event, and the system creates a tolerance window — elevated\nthresholds for the next 30 minutes. Register a recurring cron job, and the\nsystem expects that exact spike at that exact time. These aren't exceptions you\nconfigure manually. They're patterns the system learns from watching.
\nAfter a few weeks, it knows when your weekly cache warm-up runs, when your daily\nreports generate, when deploys happen. It stops bothering you about the things\nyou do on purpose.
\nCalling an LLM for every anomaly would be expensive. The trick is building\nimmune memory.
\nWhen the LLM analyses an anomaly and decides it's benign — say, a deploy spike\nor a legitimate traffic surge — that verdict gets stored. Next time the same\npattern appears, the system recognises it. No LLM call needed.
\nThis is how your security bill drops over the first few weeks. Early on,\neverything is novel. The LLM gets called constantly. A month in, most anomalies\nmatch patterns it's already seen. The LLM only gets called for genuinely new\nsituations.
\nThe more your system runs, the smarter it gets and the less it costs.
\nThe hardest part of any security tool is configuration. Getting thresholds\nright. Understanding your traffic patterns before you can tell the tool what's\nnormal.
\ndarkforest init flips this. Point it at a log sample — a day's worth of\ntraffic, a week if you've got it — and Claude reads it. Not just parsing,\nactually understanding the shape of your system. It figures out what your\nendpoints are, what normal request rates look like, what user agents show up,\nwhere your traffic comes from geographically.
Then it writes your config file for you.
\nYou review it, tweak anything that looks wrong, and you're running. No\nspreadsheets. No guesswork about what "normal" means for your specific stack.\nThe LLM that's going to watch your logs already understands them.
\nGlasswing is cool.\nOpen-source frameworks like CAI are\nmaking progress — but mostly on the offensive side, using LLMs for penetration\ntesting and vulnerability research. On the defensive side, the tooling barely\nexists. There's no open-source equivalent for the kind of adaptive monitoring\nand response I'm describing here.
\nThe building blocks are around. Centralised logging is a solved problem. Open\nstandards for security event formats are maturing. Smaller open models are more\nthan capable of pattern analysis on local infrastructure. What's missing is the\nglue — a framework that takes logs in, builds a baseline, detects anomalies, and\ncan actually respond. Something a small team can deploy without a six-figure\nsecurity budget.
\nThe threats don't discriminate by company size. The defences shouldn't either.\nThis can't be proprietary or locked behind enterprise contracts.
\nThe dark forest doesn't care how big your company is. The bots scanning your\ninfrastructure don't check your headcount before they attack. If the threats are\ngoing to be this accessible, the defences need to be too.
\nI'm building this. An open-source security agent — adaptive, autonomous, acts\nwhen something breaks the pattern. Small enough for a startup to run on their\nown infrastructure. Centralised logging, open LLMs, scoped response actions. The\npieces are all there. I'm wiring them together now.
\nFor v0.1, one real action working end-to-end: detect anomalous authentication\npatterns, call the LLM for analysis, and disable the compromised account via\nyour identity provider's API. Not just alerting — actually responding while\nyou're asleep. That's the proof of concept that matches the headline.
\nI'm actively working on this and looking for early testers. If you want alpha\naccess when it's ready, or just want to follow along,\ndrop your email below. I'll reach out when there's something to\ntry.
\n","date_published":"Sat, 11 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/llm-kills-compromised-services-at-3am.png"},{"id":"https://jonno.nz/posts/29-hours-debugging-iptables-to-boot-vms-in-4-seconds/","url":"https://jonno.nz/posts/29-hours-debugging-iptables-to-boot-vms-in-4-seconds/","title":"I Spent 29 Hours Debugging iptables to Boot VMs in 4 Seconds","content_html":"The first time I got a Firecracker VM to boot and respond to a vsock ping from\nthe host, I sat there grinning like an idiot. Typed a command on my machine, it\nreached through a kernel-level socket into a completely separate Linux system\nwith its own kernel, and got a reply. Under a second.
\nThat was about 30 hours into the project. The previous 29 were mostly fighting\nwith rootfs images and iptables rules.
\nPart 1\ncovered why I built this — Firecracker MicroVMs for running Claude Code in\nfull-permission isolation. This post is the actual build. Rootfs, networking,\nthe guest agent, and the streaming pipeline.
\nA Firecracker VM needs two things: an uncompressed Linux kernel (vmlinux, not\nbzImage — there's no bootloader) and an ext4 filesystem image to use as the\nroot disk.
The kernel is straightforward — grab a prebuilt 6.1 LTS vmlinux. The rootfs took\nmore work.
\nIt's a standard ext4 image with Debian Bookworm, and it needs everything Claude\nCode might want: Node.js 24, Python 3.11, Chromium for browser automation, git,\ncurl, jq, and the full Claude Code CLI installed globally via npm. The image\nends up at about 4GB.
\nThe guest agent — the Go binary that listens for commands from the host — lives\ninside the rootfs as a systemd service:
\nsudo mount /opt/firecracker/rootfs/base-rootfs.ext4 /mnt\nsudo cp bin/agent /mnt/usr/local/bin/agent\nsudo chmod +x /mnt/usr/local/bin/agent\n\nsudo tee /mnt/etc/systemd/system/agent.service <<'EOF'\n[Unit]\nDescription=Orchestrator Guest Agent\nAfter=network.target\n\n[Service]\nType=simple\nExecStart=/usr/local/bin/agent\nRestart=always\nRestartSec=1\n\n[Install]\nWantedBy=multi-user.target\nEOF\n\nsudo chroot /mnt systemctl enable agent.service\nsudo umount /mnt\nThat RestartSec=1 matters. If the agent crashes for any reason, systemd has it\nback up in a second. The orchestrator polls vsock every 500ms waiting for the\nagent, so even a crash during boot is barely noticeable.
You build this rootfs once, by hand. Every new VM gets a sparse copy of it.
\ninternal/vm/manager.go handles the whole lifecycle. It's sequential with\ncleanup at each step — if anything fails, it tears down what it already set up\nand returns the error.
flowchart TD\n A["Copy rootfs (sparse)"] --> B["Mount & inject network config"]\n B --> C["Create TAP device"]\n C --> D["Add iptables rules"]\n D --> E["Setup jailer chroot"]\n E --> F["Write Firecracker config JSON"]\n F --> G["Launch via jailer --daemonize"]\n G --> H["Find PID, save metadata"]\n H --> I["VM ready — poll vsock"]\nThe sparse copy is the first thing that happens:
\ncmd := exec.Command("cp", "--sparse=always", BaseRootfs, vm.RootfsPath)\n--sparse=always means zero blocks aren't allocated on disk. A 4GB image might\nonly use 2GB of actual disk space. Takes under a second on NVMe.
After copying, the rootfs gets mounted and three files are injected: a\nsystemd-networkd config with a static IP, /etc/resolv.conf for DNS, and\n/etc/hostname. Then it's unmounted and copied again into the jailer chroot.
Yeah, that's two copies of the rootfs per VM. The first for network injection,\nthe second because the jailer expects everything inside its chroot. I could\ncollapse this into one copy by injecting the network config directly into the\nchroot copy, but it's never been a bottleneck — sparse copy of 4GB takes less\ntime than Firecracker takes to boot. So I left it.
\nFirecracker's jailer is a separate binary that creates a chroot, sets up minimal\n/dev entries (kvm, net/tun, urandom), and runs the Firecracker process inside\nit. The VM config is a JSON file:
vmConfig := map[string]interface{}{\n "boot-source": map[string]interface{}{\n "kernel_image_path": "/vmlinux",\n "boot_args": "console=ttyS0 reboot=k panic=1 pci=off init=/sbin/init",\n },\n "drives": []map[string]interface{}{{\n "drive_id": "rootfs",\n "path_on_host": "/rootfs.ext4",\n "is_root_device": true,\n "is_read_only": false,\n }},\n "machine-config": map[string]interface{}{\n "vcpu_count": vm.VCPUs,\n "mem_size_mib": vm.RamMB,\n },\n "network-interfaces": []map[string]interface{}{{\n "iface_id": "eth0",\n "guest_mac": "06:00:AC:10:00:02",\n "host_dev_name": netCfg.TapDev,\n }},\n "vsock": map[string]interface{}{\n "guest_cid": vm.VsockCID,\n "uds_path": "/vsock.sock",\n },\n}\npci=off because Firecracker doesn't emulate PCI. Paths are relative to the\njailer chroot. The vsock entry creates a Unix domain socket at /vsock.sock\ninside the chroot — that's how the host talks to the guest.
Launch looks like this:
\ncmd := exec.Command(JailerBin,\n "--id", vm.JailID,\n "--exec-file", FCBin,\n "--uid", "0", "--gid", "0",\n "--cgroup-version", "2",\n "--daemonize",\n "--",\n "--config-file", "/vm-config.json",\n)\ncmd.Run()\nAfter launch there's a 2-second sleep — Firecracker needs a moment to start —\nthen the PID is found via pgrep and saved to a metadata file. If the\norchestrator restarts, it reads these metadata files and picks up where it left\noff. VMs survive orchestrator crashes.
This is where I burned the most time. Not because the concepts are hard, but\nbecause of one specific bug that had me questioning reality.
\nEach VM needs internet access for Claude Code to fetch packages, clone repos,\nand hit the Anthropic API. The approach: each VM gets a Linux TAP device on the\nhost, a dedicated /24 subnet, and iptables rules for NAT.
Subnets are deterministic, derived from the VM name using FNV-1a hashing:
\nfunc NetSlot(name string) int {\n h := fnv.New32a()\n h.Write([]byte(name))\n return int(h.Sum32()%253) + 1\n}\nVM named task-a3bfca80 might hash to slot 61, giving it subnet\n172.16.61.0/24, guest IP 172.16.61.2, TAP IP 172.16.61.1. No coordination\nneeded, no DHCP server, no IP pool to manage. The collision space is 253 slots —\nmore than enough for 12-13 concurrent VMs.
A TAP device is a virtual ethernet interface. Firecracker attaches the guest's\neth0 to it.
tap := &netlink.Tuntap{\n LinkAttrs: netlink.LinkAttrs{Name: cfg.TapDev},\n Mode: netlink.TUNTAP_MODE_TAP,\n}\nnetlink.LinkAdd(tap)\naddr, _ := netlink.ParseAddr(cfg.TapIP + "/24")\nlink, _ := netlink.LinkByName(cfg.TapDev)\nnetlink.AddrAdd(link, addr)\nnetlink.LinkSetUp(link)\nTAP names are fc-<vm-name>, truncated to 15 characters because Linux interface\nnames can't be longer. A fun constraint to discover at runtime.
Three rules per VM:
\n// NAT — rewrite source IP when traffic exits the host\nipt.AppendUnique("nat", "POSTROUTING",\n "-s", cfg.Subnet, "-o", cfg.HostIface, "-j", "MASQUERADE")\n\n// FORWARD — allow outbound from TAP\nipt.Insert("filter", "FORWARD", 1,\n "-i", cfg.TapDev, "-o", cfg.HostIface, "-j", "ACCEPT")\n\n// FORWARD — allow established/related inbound\nipt.Insert("filter", "FORWARD", 1,\n "-i", cfg.HostIface, "-o", cfg.TapDev,\n "-m", "state", "--state", "RELATED,ESTABLISHED", "-j", "ACCEPT")\nSee those Insert calls with position 1? That's the bug fix.
I originally used Append for the FORWARD rules. Traffic from the VM would\nleave the host fine (NAT worked), but return traffic got dropped. The VM could\nresolve DNS but couldn't complete TCP handshakes. I spent an embarrassing amount\nof time staring at tcpdump output before I figured it out.
Ubuntu's UFW adds a blanket DROP rule to the FORWARD chain. If you append your\nACCEPT rules, they land after UFW's DROP. They never match. The packets hit\nthe DROP rule first and get silently killed.
Insert at position 1 puts the rules before UFW's. Return traffic flows, VMs\nget internet access, everything works.
The traffic path through a working VM:
\nGuest (172.16.61.2) → eth0 → TAP (fc-task-xxx) → FORWARD ACCEPT\n→ NAT MASQUERADE (rewrite src to host IP) → host interface → internet\n→ response → RELATED,ESTABLISHED → TAP → guest eth0\nVMs can't reach each other. Each TAP device is point-to-point on its own /24.\nThere's no route between subnets.
cmd/agent/main.go — 420 lines of Go. It's a static binary that starts on boot,\nlistens on vsock port 9001, and handles five request types: ping, exec,\nwrite_files, read_file, and signal.
The interesting one is streaming exec.
\nWhen the orchestrator wants to run Claude Code, it sends an exec request with\nstream: true. The agent spawns the command, reads stdout and stderr line by\nline, and sends each line back as a framed event over the vsock connection. When\nthe process exits, it sends an exit event with the exit code.
Sounds straightforward. The tricky part is background processes.
\nClaude Code can start things that outlive the main command — dev servers, file\nwatchers, whatever it decides it needs. These child processes inherit the\nstdout/stderr pipes. If the agent waits for the pipes to close (the normal\napproach), it hangs forever because the children are still holding them open.
\nThe fix has three parts:
\n// 1. Process group isolation\ncmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}\n\n// 2. Wait for the main process, not the pipes\n<-waitDone\n\n// 3. Kill the entire process group\npgid, _ := syscall.Getpgid(cmd.Process.Pid)\nsyscall.Kill(-pgid, syscall.SIGTERM)\ntime.Sleep(500 * time.Millisecond)\nsyscall.Kill(-pgid, syscall.SIGKILL)\nSetpgid: true puts the command in its own process group. When the main process\nexits, kill the group (-pgid means "everything in this group"). SIGTERM first,\nwait half a second, then SIGKILL for anything that didn't listen.
Even after killing the group, there's a 3-second timeout waiting for the\npipe-reading goroutines to drain. If they're still stuck after that, move on and\nsend the exit event anyway. Can't let a hung pipe block the entire task.
\nThe line-by-line reader uses a 256KB buffer because Claude Code's\n--output-format stream-json can produce enormous single lines — tool results\nthat include the full contents of files it read.
Before Claude Code runs, the orchestrator writes five things into the VM via\nvsock:
\nOAuth credentials from the host's ~/.claude/.credentials.json (mode 0600). A\nsettings file that allows all tools. An environment script that sets\nCLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=true. Task metadata. And a marker file to\ncreate the output directory.
The prompt itself gets written to a temp file inside the VM to avoid shell\nescaping nightmares, then referenced in the command:
\nclaudeArgs := fmt.Sprintf(\n "claude -p \\"$(cat %s)\\" --output-format stream-json --verbose",\n promptFile,\n)\ncmd := []string{"bash", "-c",\n "source /etc/profile.d/claude.sh && " + claudeArgs}\nWhen the VM is destroyed, the rootfs — containing the credentials — is deleted.\nCredentials only exist for the lifetime of the task.
\nAfter Claude Code finishes, the orchestrator searches for files it created:
\n// Anything in the output directory\nvsock.Exec(jailID, []string{"find", outputDir, "-type", "f", "-not", "-name", ".keep"}, nil, "/root")\n\n// Any new files under /root, created after the prompt was written\nvsock.Exec(jailID, []string{"find", "/root", "-maxdepth", "2", "-type", "f",\n "-newer", "/tmp/claude-prompt.txt"}, nil, "/root")\nEach file gets downloaded via vsock.ReadFile and saved to\n/opt/firecracker/results/<task-id>/. The runner also scans the accumulated\noutput for Claude's total_cost_usd field to record what the task cost in API\ncredits.
Then the VM is destroyed. Firecracker process killed, TAP device removed,\niptables rules deleted, jailer chroot deleted, VM state directory deleted. Clean\nslate.
\nThe whole cycle — boot, inject, run, collect, destroy — typically takes 30-120\nseconds depending on how complex the prompt is. The 4-second boot and ~1-second\nteardown are rounding errors compared to the time Claude actually spends\nthinking.
\nPart 3\ngets into the fun stuff — the MCP server that lets Claude delegate tasks to\nitself, the streaming architecture, the web dashboard, and what productionising\nthis would actually look like.
\n","date_published":"Fri, 10 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/29-hours-debugging-iptables-to-boot-vms-in-4-seconds.png"},{"id":"https://jonno.nz/posts/can-you-beat-last-month/","url":"https://jonno.nz/posts/can-you-beat-last-month/","title":"Can You Beat Last Month?","content_html":"Every machine learning project needs a reality check.
\nIt's tempting to jump straight to the neural network. That's the exciting bit,\nright? But if you don't establish what a dead-simple model can do first, you've\ngot no idea whether your fancy architecture is actually learning anything useful\nor just being expensive.
\nSo before ConvLSTM gets anywhere near this data, we're going to throw three\ngloriously simple baselines at it and see how they do.
\nThe dumbest possible model. To predict April, just use March's values. Every\ncell, every crime type. Carbon copy.
\nIt sounds ridiculous, but it works surprisingly well when patterns are stable.\nAnd as we saw in the EDA, Auckland's crime hotspots are remarkably persistent.\nThe CBD doesn't suddenly go quiet. South Auckland doesn't randomly calm down.
\nOn the six-month test set (August 2025 – January 2026):
\n| Crime Type | \nMAE | \nRMSE | \n
|---|---|---|
| Theft | \n1.42 | \n3.18 | \n
| Burglary | \n0.38 | \n0.91 | \n
| Assault | \n0.22 | \n0.64 | \n
| Robbery | \n0.04 | \n0.15 | \n
| Sexual | \n0.03 | \n0.12 | \n
| Harm | \n0.01 | \n0.04 | \n
Those MAE numbers for theft and burglary look small until you remember that most\ncells are zero. For the active cells (the ones we actually care about) the error\nis larger. A busy CBD cell might have 35 thefts in one month and 28 the next.\nPersistence would be off by 7 there, which is a 20% miss on an important\nprediction.
\nInstead of copying last month, copy the same month from the previous year.\nJanuary 2026 gets predicted from January 2025. This should capture seasonal\npatterns: the summer spike, the February dip.
\nThe catch? We only have four years of data. The test set months (August–January)\neach have at most three prior examples of the same month. That's not a lot of\nseasonal training data.
\n| Crime Type | \nMAE | \nRMSE | \n
|---|---|---|
| Theft | \n1.51 | \n3.42 | \n
| Burglary | \n0.41 | \n0.97 | \n
| Assault | \n0.24 | \n0.68 | \n
| Robbery | \n0.05 | \n0.17 | \n
| Sexual | \n0.04 | \n0.13 | \n
| Harm | \n0.01 | \n0.04 | \n
Slightly worse than persistence across the board. That surprised me initially.\nShouldn't capturing seasonality help?
\nThe issue is that the 2023-to-2025 decline we spotted in the EDA bites hard\nhere. If you predict January 2026 from January 2025, you're using data from a\nperiod when crime was higher. The seasonal pattern is real, but the\nyear-over-year trend works against it. With more years of data, seasonal naive\nwould likely pull ahead.
\nFor each cell and crime type, take the average across all 36 training months.\nThis smooths out month-to-month noise and gives you a "typical" value for each\nlocation.
\n| Crime Type | \nMAE | \nRMSE | \n
|---|---|---|
| Theft | \n1.28 | \n2.95 | \n
| Burglary | \n0.35 | \n0.84 | \n
| Assault | \n0.20 | \n0.58 | \n
| Robbery | \n0.04 | \n0.14 | \n
| Sexual | \n0.03 | \n0.11 | \n
| Harm | \n0.01 | \n0.04 | \n
The best baseline. By averaging over three years, it smooths out the\nmonth-to-month noise and the year-over-year trend simultaneously. It won't\ncapture seasonal peaks or sudden changes, but for the "typical month" prediction\nit's solid.
\nYou might wonder why I'm not reporting MAPE (Mean Absolute Percentage Error).\nIt's the standard metric in a lot of forecasting work. The reason: sparse data.
\nMAPE divides the error by the actual value. When the actual value is zero (which\nit is for 91.7% of our tensor) you get division by zero. Even for cells with\nsmall counts (1 or 2 crimes), a prediction of 0 gives you 100% MAPE while a\nprediction of 2 gives you 0–100%. The metric becomes wildly unstable.
\nMAE and RMSE are more honest here. They tell you the absolute magnitude of your\nerrors in actual crime counts, which is what we care about. A miss of 3\nvictimisations means the same thing whether the cell usually has 5 or 50.
\nHere's the scoreboard going forward. Any deep learning model needs to beat the\nhistorical average baseline to justify its existence:
\n| Crime Type | \nHistorical Avg MAE | \nHistorical Avg RMSE | \n
|---|---|---|
| Theft | \n1.28 | \n2.95 | \n
| Burglary | \n0.35 | \n0.84 | \n
| Assault | \n0.20 | \n0.58 | \n
| All types | \n0.39 | \n0.95 | \n
Theft is the easiest to beat because there's the most signal: high counts, clear\nspatial patterns, strong seasonality. Robbery, sexual offences, and harm are\nessentially noise at this resolution. The models will probably predict near-zero\nfor those types and be mostly correct.
\nThe real test will be the middle ground. Can ConvLSTM or ST-ResNet predict the\nchanges in theft and burglary better than a static average? Can they catch the\nmonths where a cell spikes or dips? That's where simple baselines fall flat,\nbecause they don't model dynamics at all.
\nIf the deep learning can't meaningfully beat "just use the average," then it's\nnot worth the CPU cycles. Or in my case, the many hours of a Ryzen 5 grinding\naway.
\n","date_published":"Thu, 09 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/can-you-beat-last-month.png"},{"id":"https://jonno.nz/posts/claude-code-running-claude-code-in-4-second-disposable-vms/","url":"https://jonno.nz/posts/claude-code-running-claude-code-in-4-second-disposable-vms/","title":"Claude Code Running Claude Code in 4-Second Disposable VMs","content_html":"Running Claude Code with full permissions inside a Docker container is a\nterrible idea. I did it anyway for about a week, then built something better.
\nAnthropic has an internal platform — people have been calling it\nAntspace\nsince it got reverse-engineered from the Claude Code source — that runs AI\ncoding tasks in isolated environments. It's part of a vertical stack they're\nbuilding internally: intent goes in, code comes out, and the agent never touches\nthe host machine.
\nI wanted that. Not the whole platform-as-a-service thing, just the core idea:\ngive Claude Code a prompt, let it run with zero permission restrictions, stream\nthe output back, grab any files it created, and destroy everything when it's\ndone. On a single Linux box sitting in my office.
\nThe result is about 3,200 lines of Go and 860 lines of TypeScript. It boots a\nfresh Linux VM in ~4 seconds, runs Claude Code inside it, and tears it down when\nthe task finishes. Three ways to use it: a CLI, a REST API with a web dashboard,\nand an MCP server so Claude Code on other machines can delegate tasks to it.
\nThis first post is about why I built it this way.\nPart 2 and\nPart 3 get\ninto the actual implementation.
\nCLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=true — that's the environment variable\nthat tells Claude Code to stop asking before it runs shell commands or writes\nfiles. It just does whatever it thinks it needs to. For autonomous tasks, you\nneed this. Claude can't ask for confirmation when there's nobody watching.
The question is where you let it run.
\nDocker is the obvious first thought. Fast startup, everyone knows it, easy to\norchestrate. But containers share the host kernel. Every container on the\nmachine issues syscalls to the same Linux kernel, and a kernel vulnerability is\na vulnerability in every container on the host.\nThe isolation boundary is the container runtime,\nnot hardware — and that surface area is big.
\nFor most workloads this is fine. Running a web server in Docker? No worries. But\nrunning an AI agent that can execute arbitrary shell commands with root-level\npermissions? That's a different threat model. A container escape gives you the\nhost. And you've just given the thing inside the container permission to try\nanything.
\nAnthropic's own approach to\nsandboxing Claude Code\nuses OS-level primitives — bubblewrap on Linux, Seatbelt on macOS — for\nfilesystem and network isolation. They report an 84% reduction in permission\nprompts internally. That's smart for the normal use case where Claude is helping\nyou write code in your own project. But I wanted something more aggressive: full\nisolation where even a kernel exploit can't reach the host.
\nFirecracker is what AWS built for\nLambda and Fargate. Each MicroVM is a real KVM-backed virtual machine with its\nown guest kernel, its own memory space, and hardware-enforced isolation via\nIntel VT-x or AMD-V. The attack surface is the KVM hypervisor — which the kernel\nteam at AWS has spent years minimising.
\nThe trade-off is boot time. Containers start in under a second. Firecracker VMs\ntake about 4 seconds on my hardware once you account for the guest kernel boot,\nsystemd init, and the agent process starting up. For tasks that typically run\n20-120 seconds, 4 seconds of overhead is nothing.
\nEach VM also copies a 4GB rootfs image. Sparse copies make this fast (<1\nsecond), but it does use disk. On a machine with a 1TB NVMe, I'm not losing\nsleep over it.
\nThe hardware is an AMD Ryzen 5 5600GT with 30GB of RAM. Nothing exotic. About\n$400 worth of parts sitting under my desk. Each VM gets 2GB of RAM by default,\nso I can run roughly 12-13 VMs concurrently before the host runs out of memory.
\nThis was my favourite bit to figure out.
\nThe obvious way to communicate with a process inside a VM is SSH. Set up keys,\nopen a port, connect over the network. But SSH means key management, an open\nnetwork port inside the VM, and another service to configure. If the guest's\nnetwork breaks during a task, you've lost your control channel.
\nvsock\n(AF_VSOCK, address family 40) is a kernel-level host-guest communication\nchannel. It doesn't touch the network stack. No IP addresses, no ports, no keys.\nFirecracker exposes the guest's vsock as a Unix domain socket on the host side —\nyou connect to the socket, send CONNECT <port>\\n, and you're talking directly\nto a process inside the VM.
func Connect(jailID string, port int) (net.Conn, error) {\n socketPath := fmt.Sprintf("/srv/jailer/firecracker/%s/root/vsock.sock", jailID)\n conn, _ := net.Dial("unix", socketPath)\n conn.Write([]byte(fmt.Sprintf("CONNECT %d\\n", port)))\n // Read "OK <port>" response\n return conn, nil\n}\nOn the guest side, Go's standard library doesn't support AF_VSOCK — address\nfamily 40 doesn't exist in the net package. So the guest agent uses raw\nsyscalls:
fd, _ := syscall.Socket(40, syscall.SOCK_STREAM, 0) // AF_VSOCK = 40\n// Manually construct struct sockaddr_vm (16 bytes)\nsa := [16]byte{}\n*(*uint16)(unsafe.Pointer(&sa[0])) = 40 // family\n*(*uint32)(unsafe.Pointer(&sa[4])) = uint32(port) // port (9001)\n*(*uint32)(unsafe.Pointer(&sa[8])) = 0xFFFFFFFF // VMADDR_CID_ANY\nsyscall.RawSyscall(syscall.SYS_BIND, uintptr(fd), uintptr(unsafe.Pointer(&sa[0])), 16)\nsyscall.RawSyscall(syscall.SYS_LISTEN, uintptr(fd), 5, 0)\nYeah, that's unsafe.Pointer and manual struct layout. Not the prettiest Go\nyou'll ever write. But it works, it's fast, and the whole vsock layer is about\n160 lines shared between both binaries.
The wire protocol is dead simple — length-prefixed JSON frames:
\nfunc WriteFrame(w io.Writer, v interface{}) error {\n data, _ := json.Marshal(v)\n binary.Write(w, binary.BigEndian, uint32(len(data)))\n w.Write(data)\n return nil\n}\nEach operation (ping, exec, write files, read file) opens a new connection,\nsends one request, reads the response, and closes. Connection-per-request. Not\nfancy, but vsock connections are local and effectively instant, so there's no\nreason to complicate things with multiplexing.
\nThe whole system is two Go binaries — the orchestrator (runs on the host) and\nthe agent (runs inside each VM).
\ngraph TD\n subgraph "Host — orchestrator binary"\n API["REST API + WebSocket :8080"]\n MCP["MCP Server :8081"]\n VM["VM Manager"]\n NET["TAP + iptables"]\n TASK["Task Runner"]\n STREAM["Pub/Sub Hub"]\n VSOCK["vsock Client"]\n end\n\n subgraph "Guest — agent binary"\n AGENT["Guest Agent vsock:9001"]\n CLAUDE["Claude Code"]\n end\n\n API --> TASK\n MCP --> TASK\n TASK --> VM\n VM --> NET\n TASK --> VSOCK\n VSOCK --> AGENT\n AGENT --> CLAUDE\n TASK --> STREAM\n STREAM --> API\nThe orchestrator is a single 14MB binary with the React dashboard embedded via\n//go:embed. Copy it to a server, run it with sudo, done. Seven Go dependencies\ntotal — chi for routing, netlink for TAP devices, go-iptables for firewall\nrules, mcp-go for the MCP protocol, and a\nfew others.
The agent is a 2.5MB static binary compiled with CGO_ENABLED=0. It ships\ninside the VM's rootfs and starts via systemd on boot. Within about a second of\nthe VM coming up, the agent is listening on vsock port 9001 and ready to accept\ncommands.
They share exactly one file — internal/agent/protocol.go — which defines the\nwire protocol types and framing functions. Everything else is independent.
You give it a prompt. It does the rest.
\nFrom the CLI it looks like this:
\nsudo ./bin/orchestrator task run \\\n --prompt "Write a Python script that generates Fibonacci numbers" \\\n --ram 2048 \\\n --vcpus 2 \\\n --timeout 120\nOutput streams to your terminal in real time. When it's done:
\n=== Task Complete ===\nID: a3bfca80\nStatus: completed\nExit: 0\nCost: $0.0582\nFiles: [fibonacci.py]\nThe VM is gone. The rootfs is deleted. The TAP device and iptables rules are\ncleaned up. All that's left is the result files in\n/opt/firecracker/results/a3bfca80/.
Or you use the MCP server, and Claude Code on your laptop delegates the task to\na VM on the box under your desk. Claude spawning Claude. That bit is properly\ncool, and I'll get into it in Part 3.
\nQuick aside on this because people always ask.
\nGo produces static binaries. The agent needs to be a single file with zero\ndependencies that runs inside a minimal Debian guest — CGO_ENABLED=0 makes\nthis trivial. The orchestrator needs to manage concurrent VMs, and goroutines\nare a natural fit for that. Syscall support is first-class, which matters when\nyou're doing raw vsock operations. And it compiles in about 2 seconds, which is\nnice when you're iterating.
build-agent:\n\tCGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/agent -ldflags="-s -w" ./cmd/agent\nThat -ldflags="-s -w" strips debug info and DWARF tables, dropping the agent\nbinary from ~3.5MB to ~2.5MB. Every byte counts when you're baking it into a\nrootfs that gets copied for every VM.
Part 2 gets into\nthe actual build — the rootfs, the networking (including a fun bug with Ubuntu's\nUFW that had me staring at iptables rules for an embarrassing amount of time),\nthe guest agent, and the streaming pipeline that gets Claude's output from\ninside a VM to your browser.
\n","date_published":"Wed, 08 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/claude-code-running-claude-code-in-4-second-disposable-vms.png"},{"id":"https://jonno.nz/posts/stealing-nanoclaw-patterns-for-webapps-and-saas/","url":"https://jonno.nz/posts/stealing-nanoclaw-patterns-for-webapps-and-saas/","title":"Stealing NanoClaw Patterns for Web Apps and SaaS","content_html":"In Part 1 I pulled\napart NanoClaw's codebase and found six patterns that make an 8,000-line AI\nassistant surprisingly robust. But NanoClaw is a single-user tool running on\nyour laptop. Surely these patterns fall apart once you've got real tenants, real\nmoney, and real scale?
\nNah. Four of them translate almost directly — and the ones that don't still\nteach you something useful.
\nNanoClaw's credential proxy — where containers get a placeholder API key and a\nlocalhost proxy injects the real one — sounds like a neat trick for a personal\ntool. But this exact pattern is showing up in production Kubernetes deployments\nright now.
\nThe broader version is a\nsidecar proxy that handles credential injection\nfor any service that needs API keys or tokens. Your application code never\ntouches the real secret. A sidecar container intercepts outbound requests, swaps\nin credentials, and forwards them upstream.
\nAt Vend we managed a bunch of third-party integrations — payment gateways,\nshipping providers, accounting platforms. Each one had API keys that needed to\nlive somewhere. We went through the typical evolution: environment variables,\nthen a secrets manager, then a service that distributed keys at startup. Every\nstep was an improvement, but the keys still ended up in the application's\nmemory.
\nThe sidecar approach skips that entirely. Your app sends requests with a\nplaceholder. The proxy — which is a separate process with its own security\nboundary — does the credential swap. Even if your application gets compromised,\nthe real keys aren't there to steal.
\nIf you're running any kind of multi-service architecture where services call\nexternal APIs, this pattern is worth adopting. Your API gateway might already be\ndoing a version of it — the insight is making it explicit and consistent across\nall outbound credential flows.
\nThis is the one I keep thinking about.
\nNanoClaw uses filesystem mounts to control what each container can see. No\napplication-level permission checks — the security model is the infrastructure\ntopology. If a container can't see a file, it can't access it. No bugs, no\nmissed checks, no escalation vulnerabilities.
\nIn SaaS, we spend enormous amounts of time writing authorisation logic. Role\nchecks, permission middleware, tenant-scoping queries. And it works — until\nsomeone forgets a WHERE clause.
\nAWS's own\nSaaS tenant isolation guidance\nmakes this point explicitly: authentication and authorisation are not the same\nas isolation. The fact that a user logged in doesn't mean your system has\nachieved tenant isolation. A\nsingle missed tenant filter\non a database query and you've got a cross-tenant data leak.
\nThe NanoClaw-inspired approach is to push isolation down the stack. Separate\ndatabase schemas per tenant. Separate containers. Separate cloud accounts for\nyour highest-value customers. Not instead of application-level checks — but as a\nbackstop that catches the bugs your application-level checks inevitably have.
\nAt Xero, working across the integrations and app store teams, I saw first-hand\nhow multi-tenant data isolation gets complicated fast. The teams that had the\nfewest incidents were the ones where the infrastructure itself enforced\nboundaries, not just the application code.
\nYou don't need to go full NanoClaw and give every tenant their own container.\nBut you should be asking: if my application-level authorisation has a bug,\nwhat's my second line of defence? If the answer is "nothing" — that's the\npattern to steal.
\nNanoClaw polls SQLite every 2 seconds. No WebSockets, no event bus, no pub/sub.\nJust a loop that checks for new stuff.
\nThe instinct for most teams is to treat polling as a temporary hack you'll\nreplace with "proper" event-driven architecture later. Yan Cui wrote a\nsolid breakdown of push vs poll in event-driven systems\nand the takeaway isn't that one is always better — it's that the right choice\ndepends on your throughput, ordering, and failure-handling requirements.
\nFor a lot of internal systems, polling is the correct permanent answer.
\nAdmin dashboards. Background job status. Internal reporting. Webhook retry\nqueues. Deployment pipelines. These systems don't need sub-second latency. They\nneed reliability and simplicity. A polling loop against your database gives you\nboth, with zero infrastructure overhead.
\nAt Xero we shipped multiple times per day, and some of the internal tooling that\nsupported continuous deployment was surprisingly simple under the hood. Cron\njobs. Polling loops. SQL queries on a timer. Not because anyone was cutting\ncorners — because the requirements genuinely didn't need anything more\nsophisticated.
\nThe trap is reaching for Kafka or RabbitMQ because you think you'll need it\neventually.\n70% of startups fail due to premature scaling.\nThe infrastructure you don't deploy is the infrastructure that never breaks.
\nNanoClaw uses JSON files on the filesystem for inter-process communication.\nAtomic rename, directory-based identity, simple polling to pick up new messages.\nNo Redis. No message broker.
\nThat specific approach won't scale to a multi-tenant SaaS — but the instinct\nbehind it absolutely does. The instinct is: use the infrastructure you already\nhave.
\nFor most web apps, that means Postgres. The\nPostgres-as-queue movement\nhas been gaining serious traction, and tools like\nPGMQ make it practical. You get ACID guarantees,\nyou don't need to manage another service, and your queue is backed by the same\ndatabase you're already monitoring and backing up.
\nNanoClaw's\natomic write pattern\n— write to a temp file, rename into place — maps to INSERT INTO queue_table\nfollowed by a SELECT ... FOR UPDATE SKIP LOCKED consumer. Same principle: the\nmessage either exists completely or doesn't exist at all. No partial state.
The "just add Redis" reflex is strong in our industry. Sometimes it's the right\ncall. But I've seen plenty of teams introduce a message broker for a workload\nthat Postgres could've handled without breaking a sweat — and then spend the\nnext six months debugging consumer lag and dead letter queues.
\nThe specific techniques matter less than the discipline behind them.
\nNanoClaw's developer looked at a 500,000-line framework and asked: what are my\nactual constraints? Single user. Local machine. One AI provider. And then\nbuilt exactly the architecture those constraints required — nothing more.
\nMost teams don't do this. They build for imaginary scale, imaginary\nmulti-tenancy requirements, imaginary traffic spikes. They reach for Kubernetes\nbefore they've outgrown a single server. They deploy event buses before they've\noutgrown a polling loop. They write complex authorisation middleware before\nthey've considered whether infrastructure isolation would eliminate the problem\nentirely.
\nThe pattern worth stealing isn't the credential proxy or the polling loop or\nPostgres-as-queue. It's the habit of understanding your constraints first and\nletting them delete complexity from your architecture.
\nHardest pattern to adopt, though. Because it means admitting you're smaller than\nyou think.
\n","date_published":"Sun, 05 Apr 2026 00:00:00 GMT","image":"https://jonno.nz/og/stealing-nanoclaw-patterns-for-webapps-and-saas.png"}]}