Language tags #771
Merged
Language tags #771
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
awebson
approved these changes
May 12, 2022
There was a problem hiding this comment.
Looks great to me! Thanks so much Steve!
I think we agreed to emphasize somewhere either in the UI or in the contribution guide that the language tag should be about the languages of each prompt, not the languages of the dataset examples (which should be already documented by the datasets themselves)?
VictorSanh
approved these changes
May 12, 2022
There was a problem hiding this comment.
Looks good to me, thank you for adding this @stephenbach!
Regarding the language codes, the HF ecosystem (including datasets) is using ISO 639 for language codes (see https://huggingface.co/languages). Could we use the same here?
It will make running analysis on languages of prompts and datasets actually possible (or at least it will be smoother).
stephenbach
added a commit
that referenced
this pull request
Jul 12, 2022
* Accelerate `get_infos` by caching the `DataseInfoDict`s (#778) * accelerate `get_infos` by caching the `DataseInfoDict`s * quality * consistency * fix `filter_english_datasets` since `languages` became `language` in dataset metadatas * fix empty documents - multi_news (#793) * fix empty documents - multi_news * fix test - unrecognized variable * Language tags (#771) * Added languages widget to UI. * Style fixes. * Added English tag to existing datasets. * Add languages to viewer mode. * Update language codes. * Update CONTRIBUTING.md. * Update screenshot. * Add "Prompt" to UI to clarify languages tag usage. * Add blank languages list. Co-authored-by: Victor SANH <victorsanh@gmail.com>
stephenbach
added a commit
that referenced
this pull request
Oct 26, 2022
* remove language restrictions * add arabic dataset to primary_task * Accelerate `get_infos` by caching the `DataseInfoDict`s (#778) * accelerate `get_infos` by caching the `DataseInfoDict`s * quality * consistency * add arabic prompts * cleaning * Consistency in prompt naming. * cleaning * fix `filter_english_datasets` since `languages` became `language` in dataset metadatas * fix empty documents - multi_news (#793) * fix empty documents - multi_news * fix test - unrecognized variable * Language tags (#771) * Added languages widget to UI. * Style fixes. * Added English tag to existing datasets. * Add languages to viewer mode. * Update language codes. * Update CONTRIBUTING.md. * Update screenshot. * Add "Prompt" to UI to clarify languages tag usage. * update * update prompts * Remove duplicates lines * update * regenerate prompts * cleaning * lang tag missing Co-authored-by: Victor SANH <victorsanh@gmail.com> Co-authored-by: Stephen Bach <stephenhbach@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a language tagging feature, so that users can annotate prompts with the language(s) used in the prompt.
Even though this is motivated by the eval hackathon, this PR targets main because it affects all prompts. All existing prompts in main are tagged with English. After merging into main, another PR should merge main into eval hackathon. This will require somewhat careful coordination because new prompts on that branch will need to have their metadata updated in the .yaml before they will work in the UI.
Regarding the tags themselves, the eval group requested using the subtags in this list. I took the liberty of changing how the tags are displayed in the UI, appending the English names in parens, but this is changeable.