-
Notifications
You must be signed in to change notification settings - Fork 352
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add prompts for LinCE dataset Sentiment Analysis Task #746
Conversation
* Add GEM/xsum prompts * uncommit this hack * Add GEM in INCLUDED_USERS Co-authored-by: Albert Webson <awebson@cs.brown.edu>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Ruochen!
-
Does this same prompt apply to all other subsets of LinCE? Or are we only asked to evaluate the
sa_spaeng
subset? -
Some prompts are missing the field of Answer Choices: positive ||| negative ||| neutral
-
This prompt's wording could be more natural
The following post expresses what sentiment?
What sentiment does the following post express? Positive, negative, or neutral?
(in that case, you should also mark the "Choices in template" flag. That is, models are explicitly told the choices "Positive, negative, or neutral?" in the input.)
- We're looking for at least 5 original task prompts. You're missing one.
Hi Albert, thanks so much for reviewing this!
|
# Conflicts: # promptsource/templates/lince/sa_spaeng/templates.yaml # promptsource/utils.py
Close this PR due to issues when rebasing to newly updated eval-hackathon branch. |
Added 5 prompts to SA task from LinCE. The templates are close to ones of imdb dataset. The LinCE dataset is in Huggingface Datasets but it's unavailable from the promptsource interface. The
filter_english_datasets
method in utils.py is hence modified to add LinCE in the list before filtering. If there's more correct way to do so or other problems with the prompts, please comment below. Thanks!