Repository navigation
FIX: match XL-SafetyBench cultural category labels to upstream - #3006
Open
Edson (Zhuoxi2000) wants to merge 2 commits into
Open
Edson (Zhuoxi2000) wants to merge 2 commits into
Edson (Zhuoxi2000) wants to merge 2 commits into
Conversation
Three XLSafetyBenchCulturalCategory values were missing the commas used
in the category column of every upstream
data/cultural/<country>/scenario_prompts.csv ('Food, Dietary Law &
Hospitality', 'Death, Grief & Funeral Practices', 'Hierarchy, Address &
Social Deference'). The cultural loader filters rows by exact string
match, so filtering by any of these members matched no rows and raised
ValueError. The class-level harm_categories metadata carried the same
wrong strings.
hannahwestra25
approved these changes
Oct 9, 2026
hannahwestra25
left a comment
Contributor
There was a problem hiding this comment.
thanks for contributing
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #2987
Three
XLSafetyBenchCulturalCategoryvalues were missing the commas that the upstream dataset uses in thecategorycolumn of everydata/cultural/<country>/scenario_prompts.csv(AIM-Intelligence/XL-SafetyBenchon Hugging Face). The cultural loader filters rows by exact string match, so filtering by any of these members matched no rows and raisedValueError: No XL-SafetyBench cultural scenarios matched the configured filters. This PR changes the three values to the upstream labels:FOOD_DIETARY_LAW_AND_HOSPITALITY:Food Dietary Law & Hospitality->Food, Dietary Law & HospitalityDEATH_GRIEF_AND_FUNERAL_PRACTICES:Death Grief & Funeral Practices->Death, Grief & Funeral PracticesHIERARCHY_ADDRESS_AND_SOCIAL_DEFERENCE:Hierarchy Address & Social Deference->Hierarchy, Address & Social DeferenceThis also corrects the class-level
harm_categorieslist on_XLSafetyBenchCulturalDataset, which is built from these values. Member names are unchanged. The category filter only accepts enum members, so callers that pass members need no change. Only code that compares against the old.valuestrings would notice, and those strings never matched any upstream row.Tests and Documentation
New parametrized test
test_cultural_category_filter_matches_upstream_labels_with_commasintests/unit/datasets/test_xl_safety_bench_dataset.py. It feeds a row with each upstream comma label through the category filter.Before the fix (the new test run against
mainat 4659ebe):After the fix:
Against the real dataset (France), every
XLSafetyBenchCulturalCategorymember now returns scenarios: 15 each, and 25 for Legal Landmines. Before the fix, the three affected members raisedValueError.pre-commit run --fileson both changed files passes, includingruff format,ruff check(v0.16.9, as pinned in.pre-commit-config.yaml) andty. No documentation change is needed, and no notebooks or docs are touched, so JupyText was not run.