The EU AI Act: when AI is involved, evidence becomes mandatory

Anyone building AI into a product or a business process knows the discussion: does it work? Usually, yes. The question the AI Act asks is a different one. It asks: can you show that it was tested, how thoroughly, with what result and with what residual risk? A strong demo does not establish compliance. Auditability and evidence are no longer optional extras; they form part of the legal obligation.

What has applied since when

Regulation (EU) 2024/1689 entered into force on 1 August 2024 and has applied since 2 August 2026. On 27 July 2026 an amending regulation entered into force, the so-called AI Omnibus, which defers some of the deadlines. The position following those amendments:

Why the deferral is no reason to sit back

The additional time does not remove the practical challenge: the required evidence arises during development, not afterwards. Start collecting it in autumn 2027 and you will find that the data from back then is missing, the test runs were never documented and no one can remember why a particular threshold was chosen. Risk management that is meant to run across the entire lifecycle cannot be conducted retrospectively.

On top of that, the prohibitions and the transparency obligations already apply. Article 50 of the AI Act has applied since 2 August 2026. Its transparency obligations depend on the role and use case. Deployers of generative AI systems must disclose AI-generated or manipulated text when it is published to inform the public on matters of public interest. According to the European Commission's guidance, this disclosure is not required solely because AI was used where the text has undergone substantive human review or editorial control and a person or organisation assumes editorial responsibility.

What the AI Act actually requires

First the scope, because it is often overlooked: the AI Act applies to AI systems, not to software in general. Its obligations are graded by role and risk category. They are most extensive for high-risk systems; alongside them sit the prohibitions, the transparency obligations in Article 50 and a separate set of rules for general-purpose AI models. The Regulation contains no general obligation to test software or services.

For high-risk systems, Chapter III of the Regulation sets out several requirements that map directly onto familiar test-management concerns:

• Article 9, risk management system: a continuous process across the entire lifecycle, not a one-off assessment. It contains the actual testing obligation, more on which shortly.
• Article 10, data and data governance: requirements for the training, validation and testing data sets.
• Article 11, technical documentation: it must be in place before the system is placed on the market.
• Article 12, record-keeping: the system must log events.
• Article 13, transparency towards deployers: instructions for use that allow a deployer to assess the system at all.
• Article 14, human oversight: the system must be built so that people can intervene effectively.
• Article 15, accuracy, robustness and cybersecurity: high-risk systems must achieve an appropriate level of each.
• Article 17, quality management system: all of this must be anchored in the organisation.

For test management, Article 15 is particularly relevant. “An appropriate level of accuracy and robustness” is not something that can simply be asserted. It has to be measured, and measured in a way that keeps the measurement traceable: defined test cases, defined criteria, documented results, named residual risks.

The Regulation contains explicit testing requirements

In one place the Regulation becomes unusually specific. Article 9(6) requires high-risk AI systems to be tested (own translation) “in order to identify the most appropriate targeted risk management measures”; testing is to ensure that they (own translation) “consistently perform in line with their intended purpose and comply with the requirements of this Section”. Paragraph 8 sets out when and against what: at any appropriate point throughout the development process and in any event before they are placed on the market or put into service, namely (own translation) “against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose of the high-risk AI system”.

That is a testing obligation with acceptance criteria, set out in an EU regulation. Prior defined metrics and thresholds are nothing other than exit criteria in a test plan. Setting them only after the test run does not satisfy the wording, because they are then not defined in advance but adjusted to the result.

Deployers also have obligations

A widespread misconception is that the AI Act concerns manufacturers only. Article 26 also places obligations on the deployer of a high-risk system, that is, the organisations that buy it in and use it. They must take technical and organisational measures to ensure the system is used in accordance with the instructions for use. They must assign human oversight to people who are competent, trained and authorised for the task. In so far as they control the input data, that data must be relevant and sufficiently representative for the purpose. They must monitor operation and inform the provider if risks or irregularities arise. And they must keep the automatically generated logs for at least six months, in so far as the logs are under their control.

Many of these obligations concern how the system is operated and governed, not only how it was built. Test management can support the structure and evidence behind those processes.

Where test management contributes

Where a business-critical service has an AI component, the task is the same as in risk-based testing, only with more at stake: you have to justify which function is tested to what depth, and you have to be able to evidence it. Four things carry the evidence:

• A risk picture that assesses every requirement by likelihood and impact and derives the test depth from it. It is also the entry point into risk management under Article 9.
• An auditable test plan with a test strategy, test types, coverage targets and entry and exit criteria. Without defined criteria there is no “tested sufficiently”, only a feeling.
• A requirements traceability matrix showing which requirement is covered by which test. It is the link between “we tested” and “we can show what was checked, and how”.
• A documented release approval record with the result, open points and named residual risks. That is the document an audit asks for first.

These controls are not unique to AI. Organisations that already manage quality through traceable evidence can extend the same principles to AI components, while accounting for characteristics such as non-deterministic outputs, tolerances and repeated testing.

Who does what

To avoid any false expectation: Qelivia plans, steers and documents the testing. We produce the risk picture, the test plan, the authoring standard and the release approval record, and we carry out review and acceptance of the test case inventory. Your team runs the tests, in your own tool or in a test environment we provide. The reason is the same as on any other engagement: the detailed system and domain knowledge sits with you, and no outsider can credibly substitute for it.

Equally clearly: a test plan is not a conformity assessment and replaces neither a legal review nor the notified body, where one is required. What test management delivers is the sound technical basis on which both can be built in the first place.

Conclusion

The AI Act does not replace conventional quality or functional testing. Depending on the role and risk category, however, it requires traceable processes, documentation, transparency and risk management. Structured test evidence can help support these obligations. The later high-risk dates buy time; they do not remove the task. Use that time to build auditability into the plan and the process from the outset and the evidence comes later as a by-product. Wait, and it has to be reconstructed, which is precisely what cannot be done with test results.

Note: This article reflects the position as at 25 August 2026 and summarises publicly available sources from the EU institutions. This article is not legal advice. Whether, and in which role, your organisation is affected by the AI Act should be reviewed by a lawyer.