While many countries are still debating AI regulations, the European Union (EU) has already implemented the EU AI Act, a risk-based framework to govern AI applications. Though some details are still being worked out, AI developers must begin preparing for compliance with this law, which has started to take effect. Large language models (LLMs), which power many AI apps, are a key focus of this regulation.
To help with compliance, LatticeFlow AI, a spinout from ETH Zurich, has developed a tool called Compl-AI. This tool offers a way to evaluate LLMs and check how well they align with the EU AI Act's requirements. Compl-AI is described as the first regulation-focused benchmarking tool for AI models. It was created through a collaboration between the Swiss Federal Institute of Technology and INSAIT in Bulgaria. AI developers can use this tool to see how their models measure up against the law’s standards.
LatticeFlow has already tested popular models like Meta’s Llama and OpenAI’s GPT, publishing a compliance leaderboard that rates models from big companies like Google, OpenAI, and Anthropic. Models are scored on a scale of 0 to 1, where 1 means full compliance. The tool assesses models across 27 areas, such as avoiding harmful content, fairness, and truthfulness. Performance varies across these benchmarks, with models generally performing well in avoiding harmful instructions but struggling with fairness and reasoning.
Some areas, like copyright and privacy, are harder to evaluate because current benchmarks are limited. Despite this, the framework offers a solid start for AI compliance evaluation and will evolve as the EU AI Act continues to develop.
LatticeFlow CEO Petar Tsankov notes that most AI models have focused more on improving their capabilities than on meeting regulatory standards. He points out gaps in areas like cyberattack resilience and fairness, where many models scored poorly. Moving forward, LatticeFlow encourages researchers and developers to contribute to refining and expanding the framework, making it useful not only for the EU AI Act but for other future regulations as well.
This effort marks a key step in ensuring that AI technologies meet legal and ethical standards as AI continues to evolve.
