Skip to content

fix: faithfulness for unrelated answering #340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Nov 27, 2023
Merged
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
8b8d1fe
add langchain loaders to docs
shahules786 Oct 19, 2023
cd7f411
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Oct 20, 2023
5b18325
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Oct 26, 2023
bb8d984
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Oct 26, 2023
9cbb57d
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Oct 29, 2023
479e636
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Nov 7, 2023
3eeb7ea
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Nov 12, 2023
b09003f
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Nov 17, 2023
0d28d62
Merge branch 'main' of https://github.com/explodinggradients/ragas
shahules786 Nov 20, 2023
110cc02
reformat to json format
shahules786 Nov 20, 2023
c4036f2
add qcg to validate
shahules786 Nov 20, 2023
cf14e39
determinism experiments
shahules786 Nov 20, 2023
7ba5f46
json loader
shahules786 Nov 21, 2023
1df47bf
replace with nanmean
shahules786 Nov 21, 2023
d54ef72
move json loader
shahules786 Nov 21, 2023
3b1878d
move json loader
shahules786 Nov 21, 2023
cc128c9
fix type error
shahules786 Nov 22, 2023
24b9e25
Merge branch 'main' of https://github.com/explodinggradients/ragas in…
shahules786 Nov 22, 2023
16821c4
add error string
shahules786 Nov 22, 2023
14e7440
Merge branch 'main' of https://github.com/explodinggradients/ragas in…
shahules786 Nov 24, 2023
35fb0e6
structured output
shahules786 Nov 24, 2023
14375b7
Merge branch 'main' of https://github.com/explodinggradients/ragas in…
shahules786 Nov 26, 2023
ca04c6d
remove defaults
shahules786 Nov 26, 2023
497df5d
prompt improvements
shahules786 Nov 26, 2023
18466a2
default to None
shahules786 Nov 26, 2023
452272d
fix nli for unrelated answering
shahules786 Nov 27, 2023
a78cd49
fix nli for unrelated answering
shahules786 Nov 27, 2023
4db4f82
Update src/ragas/metrics/_faithfulness.py
shahules786 Nov 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/ragas/metrics/_faithfulness.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,17 @@
}}
]

Context:
Albert Einstein was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time.
statement_1: Nil
Answer:
[
{{
"statement_1": "Nil",
"reason": "The statement is invalid",
"verdict": "No"
}}
]


context:
Expand Down Expand Up @@ -138,6 +149,7 @@ def _score_batch(
prompts = []
for context, output in zip(contexts, result.generations):
statements = load_as_json(output[0].text).get("statements", [])
statements = statements if statements != [] else ["Nil"]
statements_str: str = "\n".join(
[f"statement_{i+1}: {st}" for i, st in enumerate(statements)]
)
Expand Down