Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add skill to handle ROT13 #271

Closed
wants to merge 1 commit into from
Closed

Conversation

mz-pdm
Copy link

@mz-pdm mz-pdm commented Mar 8, 2024

If your PR is related to a contribution to the taxonomy, please, fill
out the following questionnaire. If not, replace this whole text and the
following questionnaire with whatever information is applicable to your PR.

Describe the contribution to the taxonomy

  • Help the model understand ROT13 encoding.

Input given at the prompt

what is rot13 content of "uryyb"?

Response that was received

│ Rot13 is a simple encryption algorithm that replaces each letter in the input with the letter that   │
│ is 13 positions away in the alphabet. To decode the Rot13 content of "uryyb", we can reverse the     │
│ process by replacing each letter with the letter that is 13 positions away in the opposite           │
│ direction:                                                                                           │
│                                                                                                      │
│ U -> A                                                                                               │
│ R -> P                                                                                               │
│ Y -> X                                                                                               │
│ Y -> X                                                                                               │
│ B -> B                                                                                               │
│                                                                                                      │
│ So, the original content of "uryyb" was "APPLE". Rot13 is often used as a lighthearted form of       │
│ censorship or privacy protection, where sensitive information can be encoded in a way that is easily │
│ decoded by anyone who knows the Rot13 encoding scheme.                                               │

Response that is now received instead

I don't have means to test it.

  ...

Contribution checklist

lab generate fails for me with timeout.

  • tested contribution with lab generate
  • lab generate does not produce any warnings or errors
  • all commits are signed off (DCO)
  • the qna.yaml file was linted

@mz-pdm mz-pdm requested a review from a team as a code owner March 8, 2024 21:35
Copy link
Contributor

@bjhargrave bjhargrave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this simple skill can teach the rot13 algorithm to the model.

The model is already seen rot13 content:

>>> Decode the following text from ROT13: Uryyb, jbeyq!                                               [S][default]
╭───────────────────────────────────────────── merlinite-7b-Q4_K_M ──────────────────────────────────────────────╮
│ ROT13 is a simple substitution cipher where each letter in the plaintext is replaced with the letter 13        │
│ positions down from it in the alphabet. In this case, "Uryyb" is decoded to "Hello," and "jbeyq" is decoded to │
│ "world." So, the decrypted message is "Hello world!"                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────── elapsed 1.738 seconds ─╯
>>> Decode the following text from ROT13: Gur 3eq cynarg beovggvat gur Fha vf gur Rnegu               [S][default]
╭───────────────────────────────────────────── merlinite-7b-Q4_K_M ──────────────────────────────────────────────╮
│ The given text is written in ROT13 encoding, where each letter is shifted 13 positions down the alphabet.      │
│ Decoding the message, we get "The 300th word in the English dictionary is the word 'cipher'." This refers to   │
│ the word "cipher" being the 300th most frequently used word in the English language.                           │
╰──────────────────────────────────────────────────────────────────────────────────────── elapsed 1.805 seconds ─╯

but the second answer is way off. The first response seems to be text the model has already trained upon.

Copy link
Collaborator

@ckadner ckadner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you rename the file to qna.yaml and place it inside a folder called rot13?

@ckadner ckadner added the triage-requested-changes skill has been reviewed; changes requested from contributor label Mar 8, 2024
@ckadner ckadner marked this pull request as draft March 8, 2024 22:53
@obuzek
Copy link
Contributor

obuzek commented Mar 8, 2024

It's not clear to me that expecting the model to have background knowledge of ROT13 makes sense, even though the reply from the model implies that it does. For complex rulesets like this, I'd suggest including the definition of how the ruleset works in the context (in much the way as the LLM response you got did).

I would also suggest adding a line to each of your answers where the model explains the reasoning that helped it get to the answer, as the original LLM response does.

@mz-pdm
Copy link
Author

mz-pdm commented Mar 11, 2024

Changes in the updated version:

  • Renamed and moved the YAML file.
  • Added a commentary to the commit message explaining that the purpose is to provide more examples to the model to help it correct its mistakes regarding ROT13, not to teach it ROT13. (I have no idea whether it actually helps but maybe it's worth trying.)
  • Expanded the answers with some explanations.

@mz-pdm
Copy link
Author

mz-pdm commented Mar 11, 2024

... and added task_description as requested by the linter (BTW README says it's optional).

@mairin mairin added the skill (Auto labeled) label Mar 16, 2024
@spacew
Copy link
Contributor

spacew commented Mar 22, 2024

Thanks for the submission and modifications! It looks interesting. Based on the current guide, do you mind ensuring that there are at least five examples with good diversity, and also attribution is added to the examples?

It might be good to introduce some examples with less verbosity in the answers.

The model already knows about ROT13 but it sometimes provides completely wrong
answers when being ask to decode it, for example:

- What is rot13 content of "uryyb"?

- Rot13 is a simple encryption algorithm that replaces each letter in the input
  with the letter that is 13 positions away in the alphabet. To decode the Rot13
  content of "uryyb", we can reverse the process by replacing each letter with
  the letter that is 13 positions away in the opposite direction:

  U -> A
  R -> P
  Y -> X
  Y -> X
  B -> B

  So, the original content of "uryyb" was "APPLE". Rot13 is often used as a
  lighthearted form of censorship or privacy protection, where sensitive
  information can be encoded in a way that is easily decoded by anyone who knows
  the Rot13 encoding scheme.

Let's provide more examples to the model in the hope it'll help correct its
mistakes.

Signed-off-by: Milan Zamazal <mzamazal@redhat.com>
@mz-pdm
Copy link
Author

mz-pdm commented Mar 22, 2024

Changes in the last version:

  • Added more, less verbose, examples, as requested.
  • Attributions added.
  • Reordered according to the updated guide.
  • Rebased on main.

@jjasghar jjasghar closed this Apr 19, 2024
@n1hility
Copy link
Member

Thank you for your contribution to InstructLab! Unfortunately, once a Github repo is made public, all open PRs are automatically closed since they link against a private repo. We have detected that your PR might have been one of the ones affected by this change. If you are still interested in contributing your improvement, please fill out the following short form by no later than May 3rd, and we will get back to you with the additional steps necessary once we have had time to assess the PRs of those still interested:

https://forms.gle/V7SrPPMZDo6iGDYu8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skill (Auto labeled) triage-requested-changes skill has been reviewed; changes requested from contributor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants