AI assistants to determine if the context of the conversation

[Hugo-Calero]

If your PR is related to a contribution to the taxonomy, please, fill
out the following questionnaire. If not, replace this whole text and the
following questionnaire with whatever information is applicable to your PR.

Describe the contribution to the taxonomy

• This contribution may help to identify if a user is changing topic in a multiturn conversation.
• This is helpful in the context of building AI assistants, to decide when to keep context of the conversation or not.
• ...

Input given at the prompt

   A list of previous user queries, and the new query to compare.

Response from the original model

  ...

Response from the fine-tuned model

  ...

Contribution checklist

• The contribution was tested with ilab generate
• No errors or warnings were produced by ilab generate
• All commits are signed off (DCO)
• The qna.yaml file contains at least 5 seed_examples
• The qna.yaml file was linted and prettified (yaml-validator can do both)
• An attribution.txt file in the same folder as the qna.yaml file
• Content does not include PII or otherwise sensitive or confidential information
• Content does not include anything documented in the project's Avoid these Topics guidelines

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[mingxzhao]

You will also need to sign off on your commits as outlined here

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[mingxzhao]

@instructlab-bot precheck

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 277. The results will be presented below in the pull request status box. This may take several minutes...

[instruct-lab-bot]

Results for job ID: 277 using the model merlinite-7b!

Results can be found here.

[Hugo-Calero]

Hello @mingxzhao I can see the commits are signed off already. Eg: "Signed-off-by: Hugo Carlos Calero Díaz 112875627+Hugo-Calero@users.noreply.github.com". Is there anything else I need to do?

[mingxzhao]

Ah apologies I seemed to have missed that. If you could just update the attribution file and fix the linting issues, I can approve! It seems there are some spacing issues in your file.

[Hugo-Calero]

Hello! I have checked the listing in https://www.yamllint.com, and I see no error, is there any specific guideline I need to follow or recommended software to check?
Also, what is missing in the attribution file?
Thanks

[jjasghar]

Please resolve the linting issues.

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[Hugo-Calero]

Hi, I commited modifications to the yaml file, I hope there are no spacing issues now. Please review! Many thanks :)

[jjasghar]

Please fix the linting issue.

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[jjasghar]

@instructlab-bot precheck

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 324. The results will be presented below in the pull request status box. This may take several minutes...

[instruct-lab-bot]

Results for job ID: 324 using the model instructlab/granite-7b-lab!

Results can be found here.

[Hugo-Calero]

Hello, I see the merging is still blocked. Is there anything left to do on my side? I already fixed the linting issues I identified, I hope I didn't miss any.
Thanks

[jjasghar]

@instructlab-bot generate

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: sdg-svc. Your Job ID is 346. The results will be presented below in the pull request status box. This may take several minutes...

[jjasghar]

Looking at the precheck it seems to be very far off the rails which is good for this PR. Now the SDG generate will give us an understanding of the data that is generated and if it will add possible value to the model.

Assuming it does, then we will tag it as approved, and it will be upstreamed, then we will see from the engineering team if the model improves, only then we will merge.

[instruct-lab-bot]

Results for job ID: 346 using the model sdg service backend!

Results can be found here.

[jjasghar]

Please fix the version: 2 then we are ready the next steps.

[instruct-lab-bot]

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

• @instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
• @instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
• @instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
• @instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

[jjasghar]

@instructlab-bot precheck

[instruct-lab-bot]

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 362. The results will be presented below in the pull request status box. This may take several minutes...

[instruct-lab-bot]

Results for job ID: 362 using the model instructlab/granite-7b-lab!

Results can be found here.

[jjasghar]

With what I read through the pre-check the model already seems to do this quite well. @mingxzhao can you do a sanity check for me?
I think we should reject this as something the model already does.

[mingxzhao]

It does seem to get several of the answers wrong when compared to the user provided answers. I think this could be a good skill, but the context part of the question may need to be placed in the "context" field of the yaml for good SDG. At the moment there does seem to be several wrong answers though.

[jjasghar]

Wait really? I thought the precheck answer what we wanted. That it knew the questions didn't match the context, for instance: https://instruct-lab-bot.s3.us-east-2.amazonaws.com/precheck-pr-787-1dee610dbcdeb1f9cc76357c360aa8e63ebd1e8c-job-362/chat_2024-06-03T21_21_14.log

While the questions you've provided are related to the topic of food and dining
in Madrid, the question about the Retiro park is unrelated to the previous
conversation.

[mingxzhao]

Not quite, the question is about whether the following provided question is in the same context as the previous provided questions. The model seems to have difficulty parsing this and even answers the question separately from the context entirely.

[Hugo-Calero]

Hi, is there anything I can do to advance the status of this contribution?

[jjasghar]

We are in progress here. I believe we need this to be put in the next run, updates will be added to the PR as they arrive.

[jjasghar]

Hi! 👋
It’s been a while since you’ve seen any movement on this PR. We haven’t forgotten about you!  We’ve run into some logistical issues, hence this delay. We absolutely want your PR, and being marketed as e2e-ready is still the last stop before we get it into the upstream model.

We are thankful for your patience and ask that you please keep this PR open. As soon as we finish all our behind-the-scenes work, we’ll test the full model against your submissions and, ideally, accept your amazing contribution(s)! 

Your Community Maintainer Team.

P.S. if you have any specific questions or thoughts, don’t hesitate to comment on pull request this or email awesome@instructlab.ai and lhawthor@redhat.com, and we’ll get back to you as soon as possible.

[jjasghar]

@mcorbin-ibm where should this one live?

[mcorbin-ibm]

I think that this applies more to the "AI" side of things than "linguistics" and after doing some research in Wikipedia, in conjunction with our Dewey Decimal System reference document, I recommend placing this here in the taxonomy:

compositional_skills/technology/computer_science/ai/nlp/conversation_orchestration

Note: I did not see a context specified in the qna, so I did not put this under the compositional_skills/grounded directory, but if the qna requires a context, it should be moved there instead?