Ugur Tarlig

June 28, 2026 · 1,836 words · 9 min read

AI liability is an architecture problem

A chatbot invented medical specialist titles.

At first this sounds like a typical AI mistake: a model hallucinates, a system gives a wrong answer, someone later adds a disclaimer. The Higher Regional Court of Hamm treated the case differently.

A provider of aesthetic treatments had deployed a chatbot on its website. Users could ask questions and book appointments. When asked specific questions, the bot presented the two managing directors as specialists in certain areas of aesthetic medicine. They held no such qualifications, and two of the titles it named did not even exist in that form.

The court found the statements to be misleading commercial conduct. What matters for the architecture perspective is not only the outcome, but the attribution: the chatbot was not treated as an autonomous third party, and the statements were attributed to the operator. The ruling is not yet final, and an appeal to the Federal Court of Justice has been admitted. So it is not a definitive endpoint, and certainly not a new general “AI liability law”. But as a signal, it is enough.

Anyone who lets AI speak to the outside world in customer-facing or regulated processes has to expect that this output won’t be treated as a mere model glitch. It becomes part of the company’s communication, part of the process, part of the risk. This is exactly where liability turns from a legal question into an architecture question.

The wrong reflex: a disclaimer

The obvious reflex is a disclaimer.

This answer was generated by AI.

Please verify the information.

We accept no liability for accuracy.

Such notices can be useful, but they don’t solve the actual problem. A disclaimer doesn’t stop a system from stating a false qualification. It doesn’t stop a user from believing the statement, it doesn’t stop a chatbot from triggering an appointment, a recommendation, or a business commitment, and it doesn’t make a broken process controllable.

A disclaimer is communication.

A guardrail is architecture.

The difference matters. Communication tells the user that something can go wrong. Architecture reduces the probability, limits the damage, and makes it traceable what happened. Production AI systems need both, but treating the disclaimer as a substitute for system design shifts responsibility to the wrong place.

Output is not a side effect. It is system behavior

Many AI pilots treat output like a UI detail. The model answers, the user sees a box, maybe you tweak things. In a pilot this attitude holds, because the circle is small and nobody seriously depends on the answer.

In production, the role of the output changes. It prepares decisions, influences customers, creates medical, legal, or contractual expectations, triggers actions. The closer a system moves to a binding statement or a consequential action, the less you can rely on “the model will probably answer correctly”. At that point output is no longer text. It is the behavior of the system, and system behavior belongs in the architecture.

That also shifts the central question. Not: which model is best? But: which output is allowed to leave the building, and in which context? Almost every relevant architecture decision follows from that single question:

These are not legal footnotes. They are design constraints.

Five building blocks for attributable AI output

If AI output can be attributed to an operator, the architecture has to be built so that this output stays controllable. You don’t need a perfect, over-engineered platform, but a few building blocks should be in place before go-live.

1. Human-in-the-loop for consequential actions

Not every answer needs a human. When an internal assistant summarizes a policy or drafts a document, direct output can make sense. But when an answer is binding, contains a commitment, touches a medical context, or triggers a process, the architecture has to look different.

The human shouldn’t be stuck in the loop everywhere, or the system becomes unusable. But consequential actions need clear thresholds:

These transitions must be visible in the system. Otherwise the model implicitly decides where a process design should be deciding.

2. Output guardrails before the user

Guardrails are not just moderation filters. In production enterprise systems they check whether an answer meets the system’s rules at all:

The position in the flow is what matters: guardrails have to act before visibility, not afterwards in a dashboard. Once the user has already seen the wrong answer, the damage is no longer just technical.

3. Source grounding instead of free assertion

An AI system in a regulated environment shouldn’t simply “know”, it should show what it relies on. That sounds trivial but is architecturally demanding. A citation isn’t enough if it’s merely decorative. The answer has to come from a defensible context, and the user or reviewer has to be able to trace which part of the answer is covered by which source. Especially with RAG systems, that is the difference between:

The system sounds plausible.

and:

The system can prove why it said that.

In practice I see two particularly treacherous cases, both more dangerous than the obvious hallucination, precisely because the answer looks well-sourced.

The first case: the answer carries real source links but relies on unsuitable documents. A typical pattern is table exports. During embedding they lose their structure, and through random keyword overlaps they produce plausible but misleading hits. On top of that come documents that don’t belong in the knowledge base at all, yet were indexed anyway. The citation is then no longer evidence, just decoration.

The second, subtler case: the system combines passages from two real documents into an answer that sounds like a third, coherent document. That document never existed. In a world of controlled SOPs, where exactly one procedure corresponds to exactly one approved document, this is not merely a factual error. It is a statement that doesn’t exist in the document base at all, and yet looks citable.

Both cases lead to the same conclusion. Source grounding has to be traceable at the level of the individual statement: which sentence relies on which source? A decorative link at the end is not enough. For internal knowledge systems this is important; for customer-facing, medical, or regulatory output it is essential.

4. Audit traces instead of a chat log

A chat log is not an audit trail. For production AI systems it isn’t enough to store what the user asked and what the model answered. You have to be able to reconstruct how the answer came about:

That sounds like operations and compliance, and it is. But above all it is engineering. Without such traces you can’t analyze failures cleanly, can’t improve systems, and can’t reconstruct disputes.

5. Evals instead of hope

Many teams test AI systems the way you test demos: a few examples, a few good answers, a feeling of “it works”. That’s not enough for production. Evals are not an academic exercise, but the only reliable way to measure how often a system is wrong in the cases that matter. Domain-specific test cases are especially important:

A good eval set doesn’t cover the easiest cases, but the ones where a mistake gets expensive. If you don’t measure it yourself, someone else eventually will: a customer, an auditor, a court.

Why this belongs in regulated environments from day 1

I’ve been working on AI and GenAI systems in a regulated healthcare environment for almost two years. You learn quickly there: compliance is not a feature you add later, but a design constraint. Not because regulated organizations are hostile to innovation, but because the consequences of wrong statements are real there. A system can do more than just be wrong. It can damage trust, distort processes, or influence decisions that are supposed to stay controlled.

That’s why the order matters. Not: build the chatbot first, govern it later. But: first clarify which output is allowed, then build the system. This doesn’t mean everything has to become slow and bureaucratic. On the contrary: good architecture makes safe use faster, because it clarifies the boundaries up front.

A team doesn’t have to debate every answer if the system already knows what it may say, what it has to substantiate, what it has to block, what it has to escalate, and what it has to log. That is the difference between governance as a brake and governance as an operating system.

The model question is rarely the most important one

Many AI discussions start with the model. Claude or GPT? Cloud or open weights? A frontier model or a cheaper one? RAG or agent? These questions are relevant, but they come too early when output responsibility hasn’t been settled. It’s the same pattern I described elsewhere: the model isn’t the real bottleneck, governance, measurability, and controlled execution are.

A stronger model doesn’t fix unclear responsibility. It doesn’t replace approval boundaries, doesn’t produce an audit trail, and doesn’t decide which statement a company wants to stand behind toward a customer. The model creates possibilities. The architecture decides which of them the system is actually allowed to use.

The question before go-live

The most important sentence before the go-live of an AI agent isn’t:

How good is the model?

But:

What happens when the system is wrong in a critical case?

If the only answer to that is “then we put a disclaimer underneath”, the system is not production-ready. It becomes production-ready only when failures are expected, contained, measured, and made traceable.

That’s why AI liability is not just a legal topic. It is an architecture requirement. And the closer AI systems move to real customers, patients, and business processes, the less optional that requirement becomes.

Source: OLG Hamm, judgment of 12 May 2026, case no. 4 UKl 3/25.