As I write this, the site you are reading holds about 1,500 lines of code and 1,000 lines of styling. It runs on Astro: I have never used Astro, I have only a vague idea of how it works, and I did not write a single one of those 2,500 lines. The layout, the logic and the build configuration were all written by an AI agent, and I am perfectly happy with the result.
A good share of software work is web development, and experiences like this one have produced no shortage of commentary along the lines of “we don’t need developers any more”. It is a dangerous shortcut.
The AI did well here because this is a static site with almost no logic in it: you display formatted text, you make sure the links between pages work, and if something is wrong you can see it on the screen. Letting the model work on its own was the right call, for a reason that has nothing to do with how good it is: there was very little in this project to get wrong.
The words on this page, though, I wrote myself: the AI’s draft was… well, AI-like.
Now hand the same model the code that prices a convertible bond, or the code that checks a position against its regulatory limit, and the odds of a subtle mistake slipping in are a great deal higher.
A business made of exceptions
What makes finance different is that its rules almost always have exceptions.
A price is positive — until 20 April 2020, when an oil futures contract settled at −$37.63. So is an interest rate, until the SNB sets it at −0.75% and leaves it there for seven years. An equity trade settles T+2, except in the United States since 28 May 2024, and in Europe from October 2027. An ISIN identifies a security, but not the thing you are trading: the same ISIN can trade in New York and in Frankfurt, in two currencies, on two calendars — and, since May 2024, on two settlement cycles.
Every one of those rules holds ninety-nine times out of a hundred, and the hundredth is the one that costs you if nobody planned for it.
This is the worst possible ground for a language model. It has read the rule thousands of times and the exception almost never, so it writes the rule into the code without flagging it — a validation that rejects negative values, a type that cannot represent them.
Conversely, the existence of a rare exception does not mean it applies to the project at hand, and an agent can easily add complexity for nothing.
Plausible code is not necessarily correct
That is what makes this so tiresome. Bad AI-written code does not look bad: same shape, same sensible variable names, same structure an experienced engineer would have produced. Its mistakes are quiet ones. A date window off by a day. A nullable value handled as though it never were. It compiles, the tests pass, the build is green.
Without careful review, those mistakes pile up all the faster for every bit of autonomy you hand the model.
Elsewhere, you can live with it. In a consumer app, a bug means a button that stops responding; someone eventually complains, you fix it, you ship again. “Move fast and break things” could become a motto because in that world, breaking things is cheap.
A trading or risk system gives you no such room. A wrong number is not a broken screen; it is a position taken in error, or a limit breach that can end in financial penalties.
It is not the model’s fault
None of this says AI is oversold. But it is worth remembering what it was trained to do: produce a plausible answer, and be accommodating while doing it. It is doing its job.
Asking it to check its own correctness on top of that, in an industry where every rule drags an exception behind it, is asking a lot.
The productivity gain is real. It does not come from the freedom you give the model, but from what you build around it: controls it cannot bypass, and a team that validates what it produces.
The posts that follow describe the setup we have put in place at several clients.