In practice

Artificial Intelligence (“AI”) is playing an increasingly important role within businesses. AI is used not only for customer service, marketing and administration, but also to make decisions about people. This includes recruitment, appraisals, workforce planning, productivity measurement and even remuneration.

This raises an important question: how far may a business go in leaving decisions to algorithms?

That question is also central to recent proceedings against Uber. Drivers from the Netherlands and the United Kingdom claim that an AI-driven system partly determines how much they earn per trip. According to the claimants, there is insufficient transparency regarding the data used by the system, how remuneration is calculated and why drivers are paid differently for comparable trips. The proceedings therefore concern transparency obligations, the protection of personal data and the limits of automated decision-making.

Although this case concerns platform workers, the underlying question is relevant to virtually every business: under what conditions may an algorithm influence decisions about people’s income, appraisals or career opportunities?

The legal issue

Many business owners view AI as a tool. From a legal perspective, however, increasing attention is being paid to the consequences of decisions made with the assistance of AI.

Where algorithms are used for choices that produce legal effects or otherwise significantly affect individuals, various legal regimes may be relevant, including the General Data Protection Regulation (GDPR), employment-law standards of due care and obligations under European AI legislation.

First, the GDPR plays an important role. Organisations must be transparent about the processing of personal data, have a valid legal basis and inform data subjects about the logic and consequences of automated decision-making insofar as required by the GDPR. Additional restrictions and safeguards apply to decisions based solely on automated processing that produce legal effects or similarly significant effects.

Increasing attention is also being paid to whether those affected can understand how a decision was reached. A system that operates entirely as a “black box” may become legally and practically problematic where the outcome has major consequences for employees, job applicants or platform workers.

European AI legislation also requires organisations to take a more critical approach to AI systems used in HR, workforce and appraisal processes. In certain circumstances, such applications may be classified as high-risk, which may trigger requirements relating to risk management, data quality, transparency, human oversight and documentation.

For business owners, it is therefore important to consider not only what an AI system can technically do, but above all which decisions the system prepares or makes, which data it uses and which mechanisms for human review and accountability are in place.

Employment-law standards may also be relevant independently of the GDPR and the AI Act. Employers must act as good employers, exercise due care in appraisal and remuneration decisions, and may be held accountable where a system leads to arbitrariness, unequal treatment or outcomes that cannot be adequately reviewed.

Tips for business owners

Does your business use AI for processes that affect employees, job applicants, contractors or other persons concerned?

If so, ask yourself the following questions:

• Is it clearly documented what the AI system is used for?
• Which personal data are processed?
• Can those affected understand how a decision is reached?
• Is human oversight or review always provided?
• Can employees or other persons concerned request reconsideration of a decision?
• Have the system’s outcomes been checked for errors, bias or unintended effects?
• Does the use of AI comply with the GDPR, applicable employment-law standards and obligations under European AI legislation?

AI can help businesses work more efficiently and make better decisions. At the same time, recent discussions and proceedings show that legal responsibility does not lie with the algorithm, but with the organisation that chooses to use the system.

Transparency, due care and human oversight therefore remain essential.

Contact

Does your business use AI in HR processes, workforce planning, appraisals or other decision-making processes? Or would you like to know which obligations arise under the GDPR and European AI legislation?

Please feel free to contact ACG International. We help business owners strike the right balance between technological innovation and legal compliance.