Why AI coding can build large systems — but does not replace production engineering

Why is “writing code” not the same as “building a system”?

Because a production software system consists of more than source code. Production readiness is the sum of code, specification, tests, architecture, security, observability, deployment controls, operational processes and ownership — AI coding mainly accelerates the first of those parts.

This distinction is the core of the whole discussion. A coding agent optimises the speed at which code is produced, changed and analysed. That is enormously valuable in economic terms — and it is exactly one part out of nine.

Production readiness = code + specification + tests + architecture + security + observability + deployment controls + operational processes + ownership

What production readiness actually consists of

• Specification

  A statement, independent of the implementation, about what the system is supposed to do — and what it must never do.

• Architecture

  Decisions about boundaries, states, consistency and failure models that cannot be changed cheaply later on.

• Tests

  Not just test cases, but an independent definition of what counts as a correct result.

• Security

  The question of how the system can be pushed into doing something it should not do — a different question from functionality.

• Observability

  The ability to explain, while the system is running, what is happening right now and why.

• Deployment controls

  Approvals, staging, rollback criteria and the ability to reliably withdraw a change.

• Operational processes

  Incident response, on-call duty, recovery, maintenance windows — everything that begins after deployment.

• Ownership

  A named party accountable for architecture and operations, over years and beyond individual changes.

AI can support almost every one of these steps — generating tests, writing documentation, preparing migrations, pre-structuring reviews, analysing incidents. What it does not remove is the need for those steps. Anyone who drops the process because the code appears faster has not saved effort, but scrutiny.

What does a probabilistic development process mean?

That the same instruction can lead to different solution paths and technical decisions. It is not the delivered code that is random — once committed, AI-generated code is an entirely ordinary software artefact. What is probabilistic is the path that produced it.

This distinction decides whether the argument holds up technically. The claim that “AI code is not deterministic” is imprecise and invites contradiction. The accurate version is: the generation process is probabilistic, and the result afterwards is an artefact like any other.

Why this rarely matters for small tasks

For a single function — validating an email address, say — the solution space is manageable. If three different implementations emerge, they can be checked against each other with reasonable effort. The non-determinism is present and practically inconsequential.

Why the effect becomes relevant as systems grow

In a large system, a task does not consist of one decision but of thousands of interdependent ones. Across long chains of tasks, a probabilistic agent has to make a great many assumptions:

• Which architecture is used, and where does which piece of business logic live?
• Which data is processed synchronously, which asynchronously?
• Which errors are caught, and which states are permitted?
• How is a retry implemented, and which transaction covers which case?
• How are race conditions prevented, and what happens during partial outages?
• Which data may be cached, and which permissions apply?

The risk does not arise because more chance is involved. It arises because decisions depend on one another: the longer the chain, the greater the probability that an early wrong assumption takes effect at a much later point — where nobody is looking for it.

Non-determinism is not a reason to reject AI coding. It is a reason to industrialise verification: a probabilistic developer needs particularly strong deterministic control systems.

Why do green tests provide no guarantee?

Because implementation and tests can share the same wrong assumption when both come from the same source. A test needs more than test data; it needs an independent statement about what the correct result would be — and that is exactly what is missing when one system produces the requirement, the code and the expected test outcome.

First, because the sweeping version of this criticism is wrong: AI is very good at generating tests — unit tests, integration tests, entire suites. That is a genuine advantage, and the claim that AI-generated systems have no tests misses what actually happens in practice.

The problem arises somewhere else. It becomes critical when one and the same system interprets the requirement, writes the code, defines what correct behaviour is, writes the tests for it and then uses those tests to decide whether its own implementation was correct.

The test oracle problem

A test needs an oracle: a statement about what the result should be that is not derived from the code under test. Without it, a closed loop forms in which wrong code and wrong test confirm each other. The outcome is the most unpleasant kind of defect: every test is green and the system is wrong anyway.

How to establish an independent definition of “correct”

• Acceptance criteria taken from business specifications rather than from the code
• Contract tests between services
• Property-based testing against business invariants
• Mutation testing, to check how much the suite itself proves
• Independently written test suites and integration tests against real dependencies
• End-to-end, security, load and chaos tests
• Manual reviews of the critical invariants

A green test only proves something if the definition of “green” came into being independently of the possible defect.

Why does observability have to be designed deliberately?

Because an agent can only build the observability that is specified, provided for in the architecture or derivable from the existing system. Which signals are critical for a business is not written in the code — that is a product and architecture decision.

Here too the sweeping criticism is wrong: a coding agent can implement logs, metrics, distributed tracing, dashboards and alerts, and it often does so more cleanly than a team under time pressure. Observability is still not an automatic property of generated software.

The difference shows in the questions a production system has to be able to answer: how many requests are failing? Which dependency is causing the latency? Which customer group is affected? Which version introduced the fault? Is it local or system-wide? Which requests belong to the same transaction? None of these questions can be derived from the source code — they follow from what the business needs to know.

That requires deliberately designed logs, metrics, traces, correlation IDs, audit logs, dashboards, alerts as well as SLOs and SLIs. An agent can take on the implementation. The decision about which signal raises an alarm and which does not remains a business decision.

AI can implement observability. What it cannot do is know automatically which signals are critical for your business.

Why does security work differently from feature development?

Because software quality asks whether the system works, while security asks how someone can make it do something it should never do. That is not a matter of degree but a different question — and it is not answered by the fact that every requirement has been met.

A system can meet every functional requirement and still contain critical holes: missing authorisation, SQL injection, SSRF, unsafe deserialisation, misconfigured cloud roles, exposed secrets, missing rate limits, race conditions in permission checks, insecure dependencies.

The reason lies in the nature of the task. Feature development works against a specification; security works against an adversary who is deliberately not following that specification. An agent optimised for producing the desired function does not have that adversary in view as long as nobody introduces it explicitly.

It does not follow that AI coding has to be insecure. It follows that AI-generated code must be subject to the same security processes as code written by people — and, given the higher rate of change, rather stricter ones: static analysis, dependency checks, secret scanning, security reviews of the critical paths, penetration tests before going live.

Why does complexity grow with the size of the system?

Because large systems do not become difficult in their individual components but in how those components interact. A service can be entirely correct and still be part of a faulty overall system.

The typical sources of complexity are well known and can simply be listed: distributed systems, eventual consistency, message queues, competing writes, caches, external APIs, network errors, timeouts, retries, asynchronous jobs, multiple databases, legacy systems, permission models.

The hard part is rarely the implementation of any single one of those components. The hard part is how the overall system behaves under unusual combinations of states — and that is exactly where the example becomes concrete.

Example: a payment transaction

An agent implements the following: the user clicks “Pay”, the payment provider is called, the order is marked as paid. The happy path works, the test is green, the page opens. A production-ready system additionally has to answer:

• What happens if the payment provider responds but the database does not?
• What happens on a timeout?
• What happens if a webhook arrives twice?
• What happens if two processes work on the same order at the same time?
• What happens if a refund is triggered during a retry?
• How is idempotency ensured, and how is the state audited later?

These properties are not simply more code. They are system invariants — statements about what must hold under all circumstances. They come from a decision, not from an implementation.

Why agents reach their limits here

Coding agents need feedback in order to recognise whether a decision was right. In large systems, however, the relevant faults only appear under load, after hours or days, in rare race conditions, during partial infrastructure outages, with unusual data, with particular deployment orders or in the interplay of several services. That feedback is simply not available during development.

Why does AI accelerate good and bad engineering processes alike?

Because AI raises the rate of change, not the rate of scrutiny. A team with clean architecture, good tests, fast CI and dependable observability becomes considerably faster. A team with weak tests, unclear architecture and manual deployments produces technical debt faster than before.

A higher rate of change inevitably means more deployments, more code, more dependencies and more possible regressions. Whether that turns into speed or chaos is decided not by the tool but by the system it is used in.

That also shifts the strategic question. It is not how much code AI can write — the answer to that grows every quarter. It is: can our engineering organisation absorb the additional rate of change safely?

AI raises the rate of change. The organisation has to raise its rate of scrutiny accordingly — otherwise the only thing that grows is the gap between what is shipped and what is understood.

Why do critical IT systems need independent controls?

Because the risk does not lie in AI writing code but in the same probabilistic instance acting as producer, tester, reviewer and decision-maker at once. That creates a closed validation loop with no independent control.

Critical infrastructure — fintech, healthcare, industry, energy, public administration, insurance, identity and access management — should not rest on a process in which an agent interprets the requirements itself, decides the architecture itself, writes the code itself, defines the tests itself, assesses the results itself and then deploys to production without independent control.

That is explicitly not an argument against using it. Code suggestions, refactorings, test generation, static analysis, documentation, migrations, code reviews, incident analysis, debugging and searching large repositories are sensible uses in critical systems too. What has to be safeguarded is the release to production — through deterministic controls with organisational accountability.

Accountability cannot be delegated

Production systems live for years and require decisions that reach far beyond a single prompt: why was technology A chosen over B? What was committed to contractually? Which data may be stored? Who decides when performance and consistency have to be traded off? Who is accountable for an incident? When is a system rebuilt rather than patched further?

For critical systems it therefore has to be named who owns the architecture, who approves changes, who is accountable for security, who is on call, who decides on rollbacks and who accepts residual risk. An agent can contribute to every one of those steps — what it does not replace is a structure of accountability.

Why benchmarks do not answer the question

Coding benchmarks measure clearly bounded tasks in existing repositories, with defined test environments, limited time horizons and measurable success conditions. Production systems additionally contain contradictory and unknown requirements, stakeholder decisions, regulatory obligations, legacy systems, organisational dependencies, operations over years, incident response, real attackers and unpredictable users.

On top of that comes the reliability question. For critical IT systems, a system that completes a task correctly in 70, 80 or 90 per cent of cases is not automatically sufficient — what matters is not whether a task can be solved but with what probability a whole class of tasks is solved reliably, and how the system behaves in the remaining cases.

The relevant limit of modern coding agents is no longer capability but reliability. The agent may be part of the software lifecycle. It should not be the software lifecycle.

The target model: probabilistic generation, deterministic verification

The defensible answer is neither one hundred per cent human nor one hundred per cent autonomous. It is this: AI may work fast and creatively, and the system around it has to be strict. Quality then comes not from trusting the model but from detecting and containing errors.

In practice that means automating verification just as thoroughly as generation. A pipeline following this model looks roughly like this:

1. business requirement
2. formalised acceptance criteria
3. AI produces the implementation
4. static analysis
5. unit tests
6. contract tests
7. integration tests
8. security scans
9. check against the architecture rules
10. human or otherwise independent review
11. staging
12. observability checks
13. controlled deployment
14. automatic rollback criteria
15. monitoring in production

What stands out about this list is how little of it has to do with AI. Apart from step three, it is what good engineering has meant for years. The difference lies in the urgency: what was still manageable by hand at ten deployments a month has to be automated and demonstrable at a hundred.

AI coding solves the problem of producing code. Production engineering solves the problem of trust. The right conclusion is therefore not to ban AI coding — but to industrialise it.