Ask an AI coding agent to build the same feature twice and you'll get 2 different answers. Both might work. They'll probably name things differently, put the code in different places, and test it in different ways. Ask a third time and you'll get a third version.
If you're using AI to draft an email, that variety is harmless, and sometimes it's useful. If you're paying someone to build or change your website on a deadline and a budget, it's a real question. How does anyone deliver predictable work with a tool that gives a different answer every time?
We've spent the last couple of years working that out, first on our own projects and then on client work. The answer we keep coming back to is the tools around the AI, the ones that behave exactly the same way every time they run. Some we maintain ourselves, and the rest are well-established open-source tools that every project of ours sets up the same way.
Why AI output varies from run to run
A large language model writes code by predicting what should come next, one small piece at a time. At each step it weighs many plausible options and picks one. There's deliberate randomness in that choice, and the same prompt rarely produces the same output twice.
On top of that, the agent's answer depends on what it happens to read before it starts: which files it opened, what it found in the documentation, what was said earlier in the conversation. Models are also updated by their providers, so the same request in March and in November can be handled by quite different versions.
AI is often very good at writing code. Its output is still non-deterministic: given the same input, you can't count on the same result.
What inconsistency costs a project
On a single change, the variation is easy to live with. A developer reads the draft, tidies it up and moves on.
A whole project has hundreds of changes, and if each one is written in a slightly different style, with slightly different assumptions about where things live and how they should be tested, the codebase starts to drift. Every reviewer spends longer on each change, because nothing can be taken for granted. A fix that worked in one place gets solved differently somewhere else, and the next person to touch it has 2 patterns to understand.
That's where timelines and budgets slip. Estimates rest on knowing roughly how long a piece of work takes. When the way the work gets done changes from one run to the next, so does the time it takes, and the range on every estimate gets wider. In our experience, the speed AI brings is only worth something once that variation is under control.
Deterministic tools: the fixed points
A deterministic tool gives the same output for the same input, every time. A test suite run against the same code reports the same result. A coding standards check flags the same lines. A build script produces the same build.
When the AI varies, the checks it has to pass stay the same. We let the agent do the part where variety is fine, which is drafting the change. We let deterministic tools decide everything that has to be consistent: where the code lives, how it's structured, how it's tested, and whether it's allowed to ship.
The agent can take 3 attempts to get a change right. The gate it passes through on the third attempt is identical to the one it failed on the first. And because the checks report exactly what failed, the agent gets a precise reason to fix, and the loop settles on a working result.
Where our open-source tools fit
We've maintained open-source tooling for Drupal websites since long before AI agents arrived, mostly because we wanted our own projects to work the same way every time. It turns out that's exactly what an AI agent needs too.
Vortex: every project starts from the same shape
Vortex(Opens in a new tab/window) is our open-source Drupal project template, maintained since 2018. Every website we build starts from it, so every project has the same folder structure, the same local development setup, the same commands, the same continuous integration pipeline and the same deployment process.
For a developer joining a project, that's a big time saver. For an AI agent, it's even more useful. The agent finds the tests where tests always are, runs the same commands it ran on the last project, and gets its work checked by the same pipeline. Lessons learned on one project carry straight over to the next, because the ground under the agent hasn't moved.
The established tools inside every Vortex project
Vortex builds its checks on well-known open-source tools that most Drupal developers already know. It configures them the same way on every project, and the same pipeline runs them on every change:
| Tool | What it keeps the same |
|---|---|
| Docker(Opens in a new tab/window) | The environment the website runs in, identical on a developer's laptop, in an agent's workspace and in the CI pipeline |
| PHPUnit(Opens in a new tab/window) | The behaviour of each piece of code, checked by unit, kernel and functional tests |
| Behat(Opens in a new tab/window) | How the website behaves for visitors and editors, checked by scenarios written in plain English |
| PHPStan(Opens in a new tab/window) | Type and logic errors, caught by static analysis before the code runs |
| PHP CodeSniffer(Opens in a new tab/window) with Drupal Coder(Opens in a new tab/window) | Drupal's coding standards, applied to every line of PHP |
| Rector(Opens in a new tab/window) | Outdated code patterns, found and upgraded the same way every time |
| Twig CS Fixer(Opens in a new tab/window) | The formatting of the website's templates |
| ESLint(Opens in a new tab/window), Stylelint(Opens in a new tab/window) and Prettier(Opens in a new tab/window) | The style of JavaScript and CSS |
| Gitleaks(Opens in a new tab/window) | Passwords and keys kept out of the code, scanned on every CI run |
| Renovate(Opens in a new tab/window) | Dependency updates, raised as changes that go through the same checks as any other |
Behat Steps: tests that read the same whoever writes them
Behat, in the table above, runs scenarios written in plain English: given I am logged in as an editor, when I publish a news article, then visitors should see it on the home page. Behat Steps(Opens in a new tab/window) is our library of the ready-made steps those scenarios are built from. Version 4 ships 458 steps, and all of them follow the same 19 wording rules. A linter checks 13 of those rules automatically.
When an agent writes a test using those steps, the test reads the way any other test on the project reads, whether a person or an agent wrote it. Every step has a scenario of its own that tests it, so the building blocks are proven before the agent ever reaches for them. The result of running a scenario is a plain pass or fail, and it's the same pass or fail every time it runs against the same code.
Our smaller tools
A handful of smaller tools we maintain each pin down one more thing that would otherwise vary:
| Tool | What it keeps the same |
|---|---|
| CI runner(Opens in a new tab/window) | The same tools, at the same versions, on every CI run |
| Database data images(Opens in a new tab/window) | The same starting database in every environment, captured as container layers |
| Testmode(Opens in a new tab/window) | The listings a test checks, which show only the content the test created, so results don't depend on what editors added that week |
| Deploy Steps(Opens in a new tab/window) | The logic that runs on every deployment, as discoverable, repeatable plugins |
| PHP CodeSniffer Standard(Opens in a new tab/window) | Variable naming, checked on every change |
The naming standard is a good small example. AI agents drift between naming styles from one change to the next. A written convention asks the agent to follow it. A coding standards check fails any change that doesn't, every time, for as long as the project lives.
Together, the established tools and our own mean every change faces the same checks, whoever or whatever wrote it.
What this looks like on a project
Here's the shape of an AI-assisted change on one of our projects.
- A senior engineer defines the work and the result it needs to produce.
- The agent drafts the change inside a Vortex project, using the project's own commands and conventions.
- The agent writes or updates tests, with Behat Steps for the scenarios a visitor or editor would recognise.
- The full set of checks runs: coding standards, static analysis, unit tests and Behat scenarios, in the same pipeline every project uses.
- If anything fails, the agent gets the exact failure back and fixes it. The checks run again, unchanged.
- Once everything passes, a person reviews the change before it's merged.
The agent's draft can look different on every run. What reaches your website has passed the same checks as every other change on the project, and a person has read it.
What it means for your budget and timeline
Because the variable part of AI-assisted delivery is wrapped in fixed checks, we can offer it as a real choice on every suitable project. The same work, built to the same tested standard, comes in up to 30% lower when AI does the heavy lifting on development. Senior engineering still owns the architecture, the judgement and the review.
It also makes planning steadier. Every project starts from the same template, runs the same checks and follows the same process, so our estimates draw on a long history of very similar work. Estimates are still estimates, and on time and materials the invoice reflects the hours worked. What we can promise is that you approve scope and cost before any work starts, and a fixed-price quotation is the price.
Why our tools are open source
We depend on open source, and giving back is part of how we work. There's a practical benefit for you as well: the checks we describe in this post are public. You can read the template your website would be built on, the test steps it would be tested with, and the pipeline every change goes through, before you ever talk to us.
Every project we deliver also feeds back into these tools. When an agent trips over something, we often fix it once in the shared tooling, and every later project, ours and anyone else's, gets the fix.
The short version
AI agents give different answers to the same question. That's how they work. Deterministic tools give the same answer every time, and that's what makes them the right partner for AI. Put the agent inside a fixed template, give it tested building blocks, and run every change through the same checks, and the variation stays in the drafts, where it does no harm.