By: Catelijne Muller and Maria Rebrean

On June 14 of this year, the legislative process of the AIA entered its final phase, the Trilogue. During this phase the co-legislators (EP and Council) will negotiate the final text of the AIA under the brokerage of the European Commission. In this series of blogposts, we reflect on crucial, decisive and divisive topics for this process. After our earlier post on the ‘extra layer for high risk AI’, we now delve into the topic of ‘open-source AI components’.

A blanket exemption for Open Source AI components?

In its negotiating position on the AIA, the EP proposes to exempt open-source AI components from the scope of the AIA, as long as these are not placed on the market or put into service as part of a prohibited AI practice, a high risk AI system or a medium risk AI system. This exemption would not apply to foundation models.

We note that a decision to add a blanket exemption for open source AI (components) should not be taken lightly, for several reasons.

No clear definition of ‘open source’ or ‘open source AI components’ exists

First of all, no clear definition of ‘open source’ exists. Open source is in fact an umbrella term for a set of licences that allow users to run, copy, distribute, study, change and improve software and data, including models. The Open Source Initiative (OSI), founded in 1998, took it upon itself to draft an Open source Definition (OSD), and used it to create a list of OSI-approved licenses. In the years after, due to an increase in the number of these licences, OSI started campaigning for reducing the proliferation of open source licences.

In the meantime, the variety and number of activities that were labelled open source grew, making it ever more difficult to pinpoint what exactly is open source. This is even more important when talking of AI, or AI components. What exactly would be considered ‘open source AI components’ that would, under the proposal of the EP, be exempted from the scope of the AIA, remains equally unclear.

On June 7^th of this year, the OSI initiated “a multi-stakeholder process to define “Open Source AI”, proving that a proper definition of “open source AI” is needed but does not yet exist. Considering the complex development path of an AI system, an AI component could encompass several things, including for example source codes, data sets, models, or training processes.[1] Without a suitable definition of open source AI components, blanketly excluding them from the scope of the AIA would create legal uncertainty and even an undesired loophole that can lead to illegitimate claims of an open source status.

As the open source community expands and AI technologies rely on open source materials for their development, ensuring long lasting and effective accountability is key in shielding the market as well as health, safety and fundamental rights. As these are the exact AIA objectives, applying it (or elements of it) to open source AI components, might be the right path forward.

Upholding fundamental rights and ethical principles in open source AI components

The many known risks of AI, proven once again by the fact that the latest developments around generative AI have led to loud calls for regulation, pauses, and even developers themselves questioning their life’s work, merit ethical and legal considerations around open source. In itself, the open source movement begs a discourse of openness and democratisation of technology development and security. Yet many developers feel that it has become increasingly more difficult for them to ensure an ethical and safe use of their work.[2] Also, the same democratisation and openness can serve vile and perhaps yet unimaginable purposes. OSI describes this as: “giving everyone freedom means giving evil people freedom, too”. Indeed, open source initiatives can invite the involvement of different actors that can transpose and develop the material in undesirable settings. But also, well-intended actors may use unregulated open source AI components to create systems that turn out to be biased, erroneous, discriminatory or posing other risks to health, safety or fundamental rights. For instance, a data set that has been categorised as an exempted open source AI component may be unrepresentative, subsequently increasing the risk to the fundamental right of non-discrimination, when used for a high risk AI system.

Similar concerns around open source technology have already led to the creation of standards such as the Hippocratic License. Designed and aimed specifically at open source communities, the Hippocratic License “aims to confront the potential harms and abuses technology can have on fundamental human rights”[3] and “empowers open source communities to establish a clear set of ethical standards that licensees must abide by in order to adopt their code”.[4] Following the do-no-harm principle, the Hippocratic License defends the (re-)use open source technologies, only to the extent that it does not stimulate further risks to fundamental or human rights.[5]

Similarly, the risk-based requirements of the AI Act, as well as the EU’s Ethics Guidelines for Trustworthy AI, aim to ensure that the development and use of AI systems respect our health, safety and fundamental rights. A series of obligations that ensure e.g., data quality, transparency, accuracy, robustness, cybersecurity and the ability to exercise human oversight over the system underpin this aim. If (parts of) AI systems are developed on the basis of open source components that do not have to meet these obligations, reaching this aim might not be feasible. We argue that, without the protections of the AIA, the mitigation of the potentially harmful impact of open source technologies on health, safety or fundamental rights cannot be demanded or guaranteed.

Open source AI components: a market perspective

The EP’s introductory text to the proposed exemption (Recital (12 a) new) elaborates on the economic benefits that “open source software” has and is predicted to have on the Union market. Later, the EP writes that the exemption should apply “to foster the development and deployment of AI”. The EP’s economic justification for the exemption should however not be considered within a market vacuum, but rather within the regulated market that is pursued by the AIA.

In a recent paper, Whittaker et al. however found that: “(…) even some of the most ‘open’ AI systems do not, on their own, ensure democratic access to or meaningful competition in AI, nor does openness alone solve the problem of oversight and scrutiny”[6].

The AIA’s requirements prevent risks to rights whilst setting a quality standard for AI systems that enter the Union market, stimulating trustworthy AI innovation. If open source AI components exempted from upholding the AIA’s requirements, instead of driving innovation, they may have less competitive power in the European AI market. This is because such open source components will demand a significant (if not impossible) effort to achieve compliance, if a provider wishes to use the components for a high-risk system. From this perspective, the proposed EP exemption may not succeed in stimulating a similar economic success as earlier open source software.

Exempting open source AI components from the AIA: critical considerations

As society becomes aware of the many adverse effects of AI, there is a growing interest in the concept of open source, as it could supposedly help ensure transparency, accountability, and ethical technological development. In light of the presented concerns, i.e., lack of suitable definition, risk of creating undesired regulatory loopholes, the persistence of risks to health, safety and fundamental rights, the lack of true oversight and scrutiny possibilities, the recognition from inside the open source community that open source is not the same as a ‘free for all’, the EP’s proposed blanket exemption requires additional review.

If EU lawmakers do want to consider any exemption (or alteration) from the obligations of the AIA for open source AI components, such exemption should be subject to:

• A strict definition and categorisation of what those AI components are;
• A careful consideration of each of the AIA’s obligations vis-à-vis open source AI components to decide which of those would be acceptable and feasible to meet (potentially in an adapted form); and
• A mechanism to intervene when risks become too acute or great for the AI component to remain exempted.

At the very least, we recommend lawmakers to:

• Consider applicability of art. 4 a (new) “General principles applicable to all AI systems” (as proposed by the EP) to open source AI components (along with a proper definition and categorisation);
• Not exempt open source AI components from compliance with the prohibited practices of art. 5 AIA.

[1] Mozilla Foundation – Openness & AI: Fostering Innovation & Accountability in the EU’s AI Act

[2] Limits and Possibilities for “Ethical AI” in Open source: A Study of Deepfakes (acm.org)

[3] The Hippocratic License: An Ethical License for Open source (firstdonoharm.dev)

[4] Ibid.

[5] An Open source License That Requires Users to Do No Harm | WIRED

[6] Whittaker et al. (2023), Open (for Business): Big Tech, Concentrated Power and the Political Economy of Open AI