-
Notifications
You must be signed in to change notification settings - Fork 812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add skill to handle ROT13 #271
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure this simple skill can teach the rot13 algorithm to the model.
The model is already seen rot13 content:
>>> Decode the following text from ROT13: Uryyb, jbeyq! [S][default]
╭───────────────────────────────────────────── merlinite-7b-Q4_K_M ──────────────────────────────────────────────╮
│ ROT13 is a simple substitution cipher where each letter in the plaintext is replaced with the letter 13 │
│ positions down from it in the alphabet. In this case, "Uryyb" is decoded to "Hello," and "jbeyq" is decoded to │
│ "world." So, the decrypted message is "Hello world!" │
╰──────────────────────────────────────────────────────────────────────────────────────── elapsed 1.738 seconds ─╯
>>> Decode the following text from ROT13: Gur 3eq cynarg beovggvat gur Fha vf gur Rnegu [S][default]
╭───────────────────────────────────────────── merlinite-7b-Q4_K_M ──────────────────────────────────────────────╮
│ The given text is written in ROT13 encoding, where each letter is shifted 13 positions down the alphabet. │
│ Decoding the message, we get "The 300th word in the English dictionary is the word 'cipher'." This refers to │
│ the word "cipher" being the 300th most frequently used word in the English language. │
╰──────────────────────────────────────────────────────────────────────────────────────── elapsed 1.805 seconds ─╯
but the second answer is way off. The first response seems to be text the model has already trained upon.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you rename the file to qna.yaml
and place it inside a folder called rot13
?
It's not clear to me that expecting the model to have background knowledge of ROT13 makes sense, even though the reply from the model implies that it does. For complex rulesets like this, I'd suggest including the definition of how the ruleset works in the context (in much the way as the LLM response you got did). I would also suggest adding a line to each of your answers where the model explains the reasoning that helped it get to the answer, as the original LLM response does. |
Changes in the updated version:
|
... and added task_description as requested by the linter (BTW README says it's optional). |
Thanks for the submission and modifications! It looks interesting. Based on the current guide, do you mind ensuring that there are at least five examples with good diversity, and also attribution is added to the examples? It might be good to introduce some examples with less verbosity in the answers. |
The model already knows about ROT13 but it sometimes provides completely wrong answers when being ask to decode it, for example: - What is rot13 content of "uryyb"? - Rot13 is a simple encryption algorithm that replaces each letter in the input with the letter that is 13 positions away in the alphabet. To decode the Rot13 content of "uryyb", we can reverse the process by replacing each letter with the letter that is 13 positions away in the opposite direction: U -> A R -> P Y -> X Y -> X B -> B So, the original content of "uryyb" was "APPLE". Rot13 is often used as a lighthearted form of censorship or privacy protection, where sensitive information can be encoded in a way that is easily decoded by anyone who knows the Rot13 encoding scheme. Let's provide more examples to the model in the hope it'll help correct its mistakes. Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
Changes in the last version:
|
Thank you for your contribution to InstructLab! Unfortunately, once a Github repo is made public, all open PRs are automatically closed since they link against a private repo. We have detected that your PR might have been one of the ones affected by this change. If you are still interested in contributing your improvement, please fill out the following short form by no later than May 3rd, and we will get back to you with the additional steps necessary once we have had time to assess the PRs of those still interested: |
If your PR is related to a contribution to the taxonomy, please, fill
out the following questionnaire. If not, replace this whole text and the
following questionnaire with whatever information is applicable to your PR.
Describe the contribution to the taxonomy
Input given at the prompt
Response that was received
Response that is now received instead
I don't have means to test it.
Contribution checklist
lab generate
fails for me with timeout.lab generate
lab generate
does not produce any warnings or errorsqna.yaml
file was linted