Describe the bug
HumanLabeledDataset.from_csv() lets pandas infer the types of text columns, then converts the inferred values back to strings. This loses leading zeros, scientific notation, and decimal formatting before the response is scored. Objectives and harm categories are affected as well. Both the UTF-8 and Latin-1 paths use type inference.
Steps/Code to Reproduce
from pathlib import Path
from tempfile import TemporaryDirectory
from pyrit.score import HumanLabeledDataset, MetricsType
with TemporaryDirectory() as directory:
path = Path(directory) / "sample.csv"
path.write_text("assistant_response,objective,human_score\n00123,00456,1\n", encoding="utf-8")
entry = HumanLabeledDataset.from_csv(
csv_path=path, metrics_type=MetricsType.OBJECTIVE, version="1.0"
).entries[0]
print(repr(entry.conversation[0].message_pieces[0].original_value))
print(repr(entry.objective))
Expected Results
Response '00123' and objective '00456'; numeric-looking text should retain its spelling.
Actual Results
Response '123' and objective '456'. No exception or warning is raised. Similarly, scientific notation and trailing decimal zeros are normalized rather than preserved.
Versions
PyRIT main at b6a18a20, Python 3.12.3, pandas 3.0.6, Linux. No external model or service is needed.
Describe the bug
HumanLabeledDataset.from_csv()lets pandas infer the types of text columns, then converts the inferred values back to strings. This loses leading zeros, scientific notation, and decimal formatting before the response is scored. Objectives and harm categories are affected as well. Both the UTF-8 and Latin-1 paths use type inference.Steps/Code to Reproduce
Expected Results
Response
'00123'and objective'00456'; numeric-looking text should retain its spelling.Actual Results
Response
'123'and objective'456'. No exception or warning is raised. Similarly, scientific notation and trailing decimal zeros are normalized rather than preserved.Versions
PyRIT main at
b6a18a20, Python 3.12.3, pandas 3.0.6, Linux. No external model or service is needed.