Skip to content
This repository was archived by the owner on Sep 9, 2025. It is now read-only.

Add a skill to try to improve ROT13 generation#332

Closed
codificat wants to merge 1 commit into
instructlab:mainfrom
codificat:rot13
Closed

Add a skill to try to improve ROT13 generation#332
codificat wants to merge 1 commit into
instructlab:mainfrom
codificat:rot13

Conversation

@codificat

@codificat codificat commented Mar 11, 2024

Copy link
Copy Markdown

Describe the contribution to the taxonomy

This adds a grounded compositional editing skill to teach ROT13 to the model.

However, most current models are not capable of actually understanding and applying these instructions. Therefore, synthetic training sets generated for this skill are likely to be inaccurate.

At this time, this skill can be more useful as a benchmark / test for the process itself than as an actual skill.

Input given at the prompt

rot13 this text: "Good morning, my friend!"

Response that was received

The rot13'd text is "Grfgfher, zrffntr!"

Note that the generated rot13 text is incorrect, it corresponds to the text "Testsure, message!".

Response that is now received instead

TBD as I am not able to fine tune a model with this skill.

But here is some generated content from the skill, from lab generate with the merlinite-7b-Q4_K_M model:

[
    {
        "instruction": "Translate this sentence into English using rot13: \"Nyfb yvar vf nyfb!\"",
        "input": "",
        "output": "You are correct!"
    },
    {
        "instruction": "Explain what rot13 is and how it works.",
        "input": "",
        "output": "ROT13 is a simple letter substitution cipher that replaces a letter with the 13th letter after it in the Latin alphabet. To apply ROT13 to text, each alphabetic character in the text is replaced by the character that is 13 places further along in the alphabet, wrapping back to the beginning if necessary. For example, the letter A is replaced with N, B is replaced with O, and so on up to M, which is replaced with Z, and then the sequence continues at the beginning of the alphabet: N is replaced with A, O is replaced with B, and so on to Z, which is replaced with M. The letters that occur in the English alphabet are affected, while numbers, symbols, punctuation, whitespace, and all other characters are left unchanged. This cipher can be used for both encryption and decryption, as the same substitution pattern is used in both cases."
    },
    {
        "instruction": "Implement ROT13 on the phrase \"This is my first test!\"",
        "input": "ROT13 (or rot13, Rot13 or ROT-13) is a simple letter substitution cipher\nthat replaces a letter with the 13th letter after it in the Latin\nalphabet.\n\nApplying ROT13 to a text fragment requires examining its alphabetic\ncharacters and replacing each one by the letter 13 places further along in\nthe alphabet, wrapping back to the beginning if necessary. A becomes N, B\nbecomes O, and so on up to M, which becomes Z, then the sequence continues\nat the beginning of the alphabet: N becomes A, O becomes B, and so on to\nZ, which becomes M.\n\nTherefore, the letters are mapped as follows: A->N N->A B->O O->B C->P\nP->C D->Q Q->D E->R R->E F->S S->F G->T T->G H->U U->H I->V V->I J->W W->J\nK->X X->K L->Y Y->L M->Z Z->M\n\nOnly those letters which occur in the English alphabet are affected;\nnumbers, symbols, punctuation, whitespace, and all other characters are\nleft unchanged.\n\nFor example, the text \"Sphinx of black quartz, judge my vow.\" becomes\n\"Fcuvak bs oynpx dhnegm, whqtr zl ibj.\" when ROT13 is applied.",
        "output": "Guvf zrnaqn qvpxr!"
    },
    {
        "instruction": "ROT13 the following sentence: \"The sun shines on the meadow\"",
        "input": "ROT13 (or rot13, Rot13 or ROT-13) is a simple letter substitution cipher\nthat replaces a letter with the 13th letter after it in the Latin\nalphabet.\n\nApplying ROT13 to a text fragment requires examining its alphabetic\ncharacters and replacing each one by the letter 13 places further along in\nthe alphabet, wrapping back to the beginning if necessary. A becomes N, B\nbecomes O, and so on up to M, which becomes Z, then the sequence continues\nat the beginning of the alphabet: N becomes A, O becomes B, and so on to\nZ, which becomes M.\n\nTherefore, the letters are mapped as follows: A->N N->A B->O O->B C->P\nP->C D->Q Q->D E->R R->E F->S S->F G->T T->G H->U U->H I->V V->I J->W W->J\nK->X X->K L->Y Y->L M->Z Z->M\n\nOnly those letters which occur in the English alphabet are affected;\nnumbers, symbols, punctuation, whitespace, and all other characters are\nleft unchanged.\n\nFor example, the text \"Sphinx of black quartz, judge my vow.\" becomes\n\"Fcuvak bs oynpx dhnegm, whqtr zl ibj.\" when ROT13 is applied.",
        "output": "Gur svthr vf n pbzzr, gur zrffntr gb ybfg vf abj."
    },
    {
        "instruction": "Tell me if the following text is already ROT13 encoded or not: \"Gur dhvpx oebja sbk whzcf bire gur ynml qbt\"",
        "input": "The input is the same as the previous instruction, as there is no need for a specific example for this type of instruction.",
        "output": "Yes, the text is already ROT13 encoded."
    },
    {
        "instruction": "Provide an example of text that is ROT13 encoded using the alphabet mapping provided earlier",
        "input": "The rot13 skill enables encoding text with ROT13, which replaces each letter in the alphabet with the letter 13 places further along in the alphabet, wrapping back to the beginning if necessary. The letters are mapped as follows: A->N N->A B->O O->B C->P P->C D->Q Q->D E->R R->E F->S S->F G->T T->G H->U U->H I->V V->I J->W W->J K->X X->K L->Y Y->L M->Z Z->M",
        "output": "The rot13 skill enables encoding text with ROT13, which replaces each letter in the alphabet with the letter 13 places further along in the alphabet, wrapping back to the beginning if necessary. The letters are mapped as follows: A->N N->A B->O"
    }
]

Contribution checklist

This adds a grounded compositional editing skill to teach ROT13 to the model.
It is grounded because instructions/context about what ROT13 is and how to apply
it are provided as `context` in each example.

However, most current models are not capable of actually understanding and
applying these instructions. Therefore, synthetic training sets generated for
this skill are likely to be inaccurate.

At this time, this skill can be more useful as a benchmark / test for the
process itself than as an actual skill.

Signed-off-by: Pep Turró Mauri <pep@redhat.com>
@codificat codificat requested a review from a team March 11, 2024 12:15
@codificat codificat changed the title Add a skill for ROT13 Add a skill to try to improve ROT13 generation Mar 11, 2024
@codificat

Copy link
Copy Markdown
Author

Note that there is another existing PR around ROT13, #271.

The goals are different though: the objective in the other PR is to identify (and decode) ROT13 fragments, while this one here is about applying rot13 itself.

I believe #271 has better chances of succeeding as a skill at the moment, in particular to identify / extract ROT13 fragments, but that the actual application of the algorithm will fail.

@mairin mairin added the skill (Auto labeled) label Mar 16, 2024
@obuzek

obuzek commented Mar 22, 2024

Copy link
Copy Markdown
Contributor

Since #271 exists, can you combine forces with @mz-pdm and make a combined suggestion? It's not likely that we need two skills to teach ROT13. Adding variations of ways to approach the problem in a single skill is appropriate.

thank you!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

skill (Auto labeled) triage-rejected PR fails to meet criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants