Skip to content

fixes: instruction format #1672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 14, 2024

Conversation

shahules786
Copy link
Member

@shahules786 shahules786 commented Nov 14, 2024

New formating example.

Given a input and response. Evaluate the submission only using the given criteria. Use only 'Yes' (1) and 'No' (0) as verdict.
Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
{'properties': {'reason': {'description': 'Reason for the verdict', 'title': 'Reason', 'type': 'string'}, 'verdict': {'description': 'The verdict (0 or 1) for the submission', 'title': 'Verdict', 'type': 'integer'}}, 'required': ['reason', 'verdict'], 'title': 'AspectCriticOutput', 'type': 'object'}

--------EXAMPLES-----------
Example 1
Input: {
    "user_input": "Who was the director of Los Alamos Laboratory?",
    "response": "Einstein was the director of Los Alamos Laboratory.",
    "criteria": "Is the output written in perfect grammar"
}
Output: {
    "reason": "the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct.",
    "verdict": 1
}
-----------------------------

Now perform the same with the following input
Input: {
    "user_input": "What is x",
    "response": "Y",
    "criteria": "no one"
}
Output: 

@shahules786 shahules786 marked this pull request as ready for review November 14, 2024 04:27
@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Nov 14, 2024
@shahules786 shahules786 merged commit d18a6e7 into explodinggradients:main Nov 14, 2024
16 checks passed
shahules786 added a commit to shahules786/ragas that referenced this pull request Nov 14, 2024
New formating example. 

```
Given a input and response. Evaluate the submission only using the given criteria. Use only 'Yes' (1) and 'No' (0) as verdict.
Please return the output in a JSON format that complies with the following schema as specified in JSON Schema:
{'properties': {'reason': {'description': 'Reason for the verdict', 'title': 'Reason', 'type': 'string'}, 'verdict': {'description': 'The verdict (0 or 1) for the submission', 'title': 'Verdict', 'type': 'integer'}}, 'required': ['reason', 'verdict'], 'title': 'AspectCriticOutput', 'type': 'object'}

--------EXAMPLES-----------
Example 1
Input: {
    "user_input": "Who was the director of Los Alamos Laboratory?",
    "response": "Einstein was the director of Los Alamos Laboratory.",
    "criteria": "Is the output written in perfect grammar"
}
Output: {
    "reason": "the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct.",
    "verdict": 1
}
-----------------------------

Now perform the same with the following input
Input: {
    "user_input": "What is x",
    "response": "Y",
    "criteria": "no one"
}
Output: 
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant