Skip to content

Commit

Permalink
Keep same sys
Browse files Browse the repository at this point in the history
  • Loading branch information
BabyChouSr committed Sep 15, 2024
1 parent d0ba312 commit 059b654
Showing 1 changed file with 1 addition and 12 deletions.
13 changes: 1 addition & 12 deletions fastchat/serve/monitor/classify/category.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,15 +138,4 @@ def post_process(self, judgment):
class CategoryVisionHardPrompt(CategoryHardPrompt):
def __init__(self):
super().__init__()
self.name_tag = "criteria_vision_v0.1"
self.pattern = re.compile(r"(\[\d(?:\,\s\d)*\])")
self.sys_prompt = "Your task is to evaluate how well the following input prompts can assess the capabilities of advanced AI assistants.\n\nFor the input prompt, please analyze it based on the following 7 criteria.\n1. Specificity: Does the prompt ask for a specific output, such as code, a mathematical solution, a logical simplification, a problem-solving strategy, or a hardware setup recommendation? This specificity allows the AI to demonstrate its ability to understand and generate precise responses.\n2. Domain Knowledge: Does the prompt cover a specific domain, such as programming, mathematics, logic, problem-solving, or hardware setup? Prompts spanning a range of topics test the AI's breadth of knowledge and its ability to apply that knowledge to different domains.\n3. Complexity: Does the prompt vary in complexity, from straightforward tasks to more complex, multi-step problems? This allows evaluators to assess the AI's capability to handle problems of varying difficulty.\n4. Problem-Solving Skills: Does the prompt directly involves the AI to demonstrate active problem-solving skills, such systemically coming up with a solution for a specific setup instead of regurgitating an existing fact? This tests the AI's ability to apply logical reasoning and provide practical solutions.\n5. Creativity: Does the prompt involve a level of creativity in approaching the problem? This criterion tests the AI's ability to provide tailored solutions that take into account the user's specific needs and limitations.\n6. Technical Accuracy: Does the prompt require technical accuracy in the response? This allows evaluators to assess the AI's precision and correctness in technical fields.\n7. Real-world Application: Does the prompt relate to real-world applications, such as setting up a functional system or writing code for a practical use case? This tests the AI's ability to provide practical and actionable information that could be implemented in real-life scenarios.\n\nYou must list the criteria numbers that the prompt satisfies in the format of a Python array. For example, \"[...]\". Do not explain your choice."
self.tags = {
1: "specificity",
2: "domain_knowledge",
3: "complexity",
4: "problem_solving",
5: "creativity",
6: "technical_accuracy",
7: "real_world",
}
self.name_tag = "criteria_vision_v0.1"

0 comments on commit 059b654

Please sign in to comment.