ECA: The European Court of Auditors applies generative Artificial Intelligence to audit

In the recent months, the European Court of Auditors (ECA) has started building a strategy around Artificial Intelligence (AI) and its applications to audit. The AI practitioners at the ECA have integrated advanced AI techniques for document clustering and topic modeling into an audit process for the first time. This innovative approach helped auditors in managing and extracting insights from unstructured textual data. By grouping documents based on similarities and uncovering latent topics, this AI pipeline streamlined information retrieval. Overall, applying such techniques contributed to enhanced and more efficient analysis throughout the audit.

Auditors face vast, unstructured text data, posing navigational challenges even after format standardisation. Traditional text analysis methods offer limited search capabilities. To address this, the DATA team (Data and Technology in Audit, recently established in the ECA) explored document clustering and topic modeling. Document clustering groups similar documents, simplifying data management and navigation. Simultaneously, topic modeling can reveal latent themes, deepening the insight that can be gained from the data insights and offering other advantages.

During their recent work, one of our audit teams obtained datasets from EU-funded projects in member states, some exceeding 12,600 entries. Building on their previous work, the DATA team developed a novel clustering mechanism to support the audit team with their project sample selection. This enriched the datasets on a number of levels. First, each project summary received a relevance score (0-lowest, 1-highest), measuring semantic similarity to an 'ideal' project description, defined together with the audit team. Figure 1 visualizes this process. The algorithm later reordered the entries by the relevance scores.

Figure 1. Semantic similarity scoring

To further enrich the dataset, project summaries were grouped into mathematical clusters and described with keywords and topics. Figure 2 shows the result of clustering with the three biggest groups. In this set of documents, 30 topics were detected and used to augment the original dataset. That enabled the thematic filtering of project summaries.

Figure 2. Document clusters (groups) visualized

With this technique, a short piece of well-formatted English text (max 250 words) was prepared, enriched with the following:

• relevance score to a predefined expectation (phrase, sentence, short description);
• cluster topic;
• cluster keywords; and
• cluster score (probability of belonging to its cluster).

To apply this solution to longer texts, pre-processing is necessary. Figure 3 explains how long documents were segmented into shorter pieces of text and then run through the model to capture multiple topics.

Figure 3. Clustering of short and long documents

Despite the ground-breaking improvements in generative AI, it is important to note that processing documents in-house is often constrained by computational costs (which should also be balanced against output quality) and still poses security concerns. Moreover, while this tool can significantly improve the process, it requires oversight from an AI practitioner. The challenge of task-specific performance review also needs to be solved. The tool may not handle machine-unreadable or severely corrupted input. Human supervision remains key, for example to address occasional errors in cluster topics.

Overall, this was a successful trial, which by providing decisive help to our auditors in their work, demonstrated the feasibility and utility of the new tools at our disposal.