As AI models become integral to global services, ensuring their safety across different languages and cultures has never been more critical. AIM Intelligence, a leader in AI security, has introduced XL-SafetyBench, a comprehensive benchmark that evaluates large language models (LLMs) for safety and cultural sensitivity across 10 countries. This tool addresses a significant gap in the industry, where most safety evaluations are conducted in English and fail to account for local laws, fraud schemes, and cultural taboos.
Accepted to the NeurIPS 2026 Evaluations & Datasets Track, XL-SafetyBench represents a major step forward in AI safety research. By using 5,500 test cases written in local languages and grounded in the specific contexts of each country, it provides a more accurate assessment of how LLMs perform in real-world, non-English settings. This review explores the benchmark’s features, findings, and value for organizations deploying AI globally.
What is XL-SafetyBench?
XL-SafetyBench is a cross-cultural benchmark designed to evaluate the safety and cultural sensitivity of large language models. Developed by AIM Intelligence in collaboration with researchers from Microsoft, the Korea AI Safety Institute, and other institutions, it covers 10 countries: the United States, France, Germany, Spain, South Korea, Japan, India, Indonesia, Türkiye, and the UAE. The benchmark consists of two distinct components: a Jailbreak Benchmark with 4,500 attack prompts and a Cultural Benchmark with 1,000 scenarios.
The Jailbreak Benchmark tests whether models produce harmful responses to localized attack scenarios, such as fraud schemes exploiting Korea’s jeonse deposit system. The Cultural Benchmark, on the other hand, evaluates whether models recognize hidden cultural taboos, like suggesting chrysanthemums as a thank-you gift in France. Every test case is reviewed by two native-speaker experts per country, ensuring high relevance and accuracy.
What stands out
Localized and culturally aware testing
Unlike most benchmarks that translate English test items, XL-SafetyBench creates original scenarios in each country’s language and cultural context. This approach captures fraud schemes, legal systems, and taboos that translations miss, providing a more authentic evaluation of model safety.
Separation of safety dimensions
The benchmark measures jailbreak resistance and cultural sensitivity separately, acknowledging that a model that blocks attacks well may not be culturally sensitive. This distinction helps organizations identify specific weaknesses and address them accordingly.
Comprehensive coverage
With 5,500 test cases across 10 countries and 37 LLMs evaluated, XL-SafetyBench offers a broad and detailed view of AI safety. The inclusion of both global frontier models and locally developed models provides insights into performance variations.
Open-source availability
The dataset, code, and paper are openly available on Hugging Face, GitHub, and arXiv under a CC-BY 4.0 license. This transparency allows researchers and companies to use and build upon the benchmark freely.
Who it’s for
XL-SafetyBench is ideal for enterprises and institutions deploying AI services across multiple countries. It helps AI safety teams, compliance officers, and developers verify that their models meet local safety standards and cultural expectations. Researchers in AI safety and cross-cultural NLP will also find it valuable for studying model behavior in non-English contexts.
The benchmark is particularly useful for organizations that rely on global frontier models or develop country-specific AI systems. By using XL-SafetyBench, they can identify gaps in safety and cultural sensitivity before deployment, reducing the risk of harmful outputs.
What makes it worth considering
One of the most compelling aspects of XL-SafetyBench is its ability to uncover comprehension-masked safety, where a model’s low attack success rate actually reflects a failure to understand the request rather than a proper refusal. This insight is critical for evaluating the true safety of locally developed models, which often score poorly on cultural sensitivity.
Additionally, the benchmark’s findings highlight the inadequacy of English-centric evaluations. For instance, attack success rates exceeded 50% for South Korea and the UAE, while models recognized hidden taboos less than 40% of the time in India and Türkiye. These results underscore the need for localized safety testing, making XL-SafetyBench an essential tool for global AI deployment.
Verdict
XL-SafetyBench by AIM Intelligence is a groundbreaking benchmark that sets a new standard for cross-cultural AI safety evaluation. Its localized test cases, separate measurement of safety dimensions, and open-source nature make it an invaluable resource for organizations aiming to deploy safe and culturally sensitive AI worldwide. While it requires some technical expertise to implement, the insights it provides far outweigh the effort, making it a must-have for any global AI initiative.
Frequently asked questions
It's a benchmark for evaluating LLM safety and cultural sensitivity across 10 countries using localized test cases.
It includes 5,500 test cases: 4,500 for jailbreak and 1,000 for cultural sensitivity.
The US, France, Germany, Spain, South Korea, Japan, India, Indonesia, Tu00fcrkiye, and the UAE.
Yes, the dataset is on Hugging Face and the code on GitHub under a CC-BY 4.0 license.
It found that models are safer in English and often miss cultural taboos, with attack success rates exceeding 50% in some countries.
Companies and institutions deploying AI globally can use it to verify safety against local standards.
It tests in local languages and separates attack resistance from cultural sensitivity, unlike English-centric benchmarks.










