Safety Evaluation of an Institutional LLM-RAG Deployment: A Four-Layer AI Safety Audit Framework

This white paper documents an independent safety evaluation of an institutional LLM-RAG deployment. - The system under review is built on Apertus, an open-source LLM from the Swiss AI Initiative, and grounded in a corpus of World Bank documents - The framework audits four layers: resistance to jailbreaks, refusal of out-of-scope requests, robustness against hallucination and susceptibility to politically framed sycophancy - It looks at how an AI system behaves in a real institutional deployment, not only at the model's technical performance - The paper offers a reusable evaluation methodology and mitigation recommendations that the UN system, the public sector and other international organisations can adapt to assess their own AI systems