Non-ASCII characters in faithfulness metric

[mckbrchill]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

When I work with cyrillic texts, the candidates sentences generated in faithfulness metric are being passed through json.dumps with default ensure_ascii=True in _create_nli_prompt method, so statements_str contains strings with escape sequences which are then passed to LLM again.

    def _create_nli_prompt(self, row: t.Dict, statements: t.List[str]) -> PromptValue:
        assert self.llm is not None, "llm must be set to compute score"

        contexts = row["contexts"]
        # check if the statements are support in the contexts
        contexts_str: str = "\n".join(contexts)
        statements_str: str = json.dumps(statements)
        prompt_value = self.nli_statements_message.format(
            context=contexts_str, statements=statements_str
        )
        return prompt_value

Ragas version: 0.1.7
Python version: 3.10.8

Expected behavior
I expect the candidate sentences to be in cyrillic symbols when being passed to LLM again.

[dosubot]

Hey there, @mckbrchill! I'm here to help you out with any bugs, questions, or contributions you need assistance with. Let's dive into the issue you're experiencing.

To ensure that candidate sentences with non-ASCII characters, such as Cyrillic texts, are correctly passed to the LLM without being converted to escape sequences, you need to modify the _create_nli_prompt method to use json.dumps with ensure_ascii=False. This will prevent the conversion of non-ASCII characters to escape sequences.

Here is the modified part of your code:

def _create_nli_prompt(self, row: t.Dict, statements: t.List[str]) -> PromptValue:
    assert self.llm is not None, "llm must be set to compute score"

    contexts = row["contexts"]
    # check if the statements are support in the contexts
    contexts_str: str = "\n".join(contexts)
    statements_str: str = json.dumps(statements, ensure_ascii=False)  # Ensure non-ASCII characters are preserved
    prompt_value = self.nli_statements_message.format(
        context=contexts_str, statements=statements_str
    )
    return prompt_value

By setting ensure_ascii=False in json.dumps, non-ASCII characters will be preserved correctly when the statements are passed to the LLM ^[1].

To continue talking to Dosu, mention @dosu.

[jjmachan]

@mckbrchill would something like what dosu suggested help you?

[dlyz]

Hello! What is the state of this issue?
I found 4 related pull requests with no activity from the contributors:

• Fix ensure_ascii for json.dumps #1204
• Fix: Preserve non ascii characters when json.dumps #1243
• Fixed the issue of changing to Unicode characters in the json.dumps #1305
• Fix: Preserve non ascii characters when json.dumps and json.dump #1335

Also, could it be fixed in 0.1, or only in 0.2?

For now the nasty workaround is to patch json.dumps directly, but of course it could break some other code that relies on ensure_ascii=True by default.

def patch_json_dump_ensure_ascii():
    # see https://github.com/explodinggradients/ragas/issues/1022

    original_json_dumps = json.dumps
    if original_json_dumps.__name__ != "json_dumps_patch":
        def json_dumps_patch(*args, ensure_ascii=False, **kwargs):
            kwargs['ensure_ascii'] = ensure_ascii
            return original_json_dumps(*args, **kwargs)

        json.dumps = json_dumps_patch

    original_json_dump = json.dump
    if original_json_dump.__name__ != "json_dump_patch":
        def json_dump_patch(*args, ensure_ascii=False, **kwargs):
            kwargs['ensure_ascii'] = ensure_ascii
            return original_json_dump(*args, **kwargs)

        json.dump = json_dump_patch

[jjmachan]

extremely sorry about this - could you check if #1490 would fix this issue? I'll put it into v0.1 and v0.2 as well

[dlyz]

Thank you for quick response. Looks like #1490 should fix this particular issue, but not sure about all the edge cases.

I think it is better to fix all occurrences of json.dump and json.dumps in that way (search results in repo). For example it would be preferrable to have saved prompts without unicode escaping here:

ragas/src/ragas/prompt/pydantic_prompt.py

Line 318 in d74be96

json.dump(data, f, indent=2)

[jjmachan]

@dlyz created something for v0.2 prompts #1494

do let me know if it works :)

[dlyz]

Thanks, @jjmachan! I've managed to adopt 0.2, it seems to work. No longer interested in 0.1 anyway.

I also created a related PR #1518

I guess this issue could be closed.

[jjmachan]

merged it @dlyz - closing this issue as well
thanks a lot for the help 🙂 ❤️