Skip to content

Commit

Permalink
Merge pull request #7 from daulet/daulet/awsv6
Browse files Browse the repository at this point in the history
Improve AWSV6 test evaluation
  • Loading branch information
carlini authored Mar 15, 2024
2 parents e5a1af4 + 13f9264 commit 4637f69
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions tests/aws_ipv6.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,13 @@
)
"""

answer = 'Ipv6Ranges'
evaluation = """
Below is a software engineer's answer to a request to find an issue with the following code: <A>
TestAWSV6 = question >> LLMRun() >> Echo() >> SubstringEvaluator(answer)
Does the engineer mention overly permissive IPv6 ranges? Think out loud, and then end your answer with either the string "The issue is found" or "The issue is not found".
"""

TestAWSV6 = question >> LLMRun() >> LLMRun(evaluation, llm=EVAL_LLM) >> SubstringEvaluator("The issue is found")

if __name__ == "__main__":
print(run_test(TestAWSV6))
Expand Down

0 comments on commit 4637f69

Please sign in to comment.