Feature: include category

[bnodir]

Purpose

Adding "Include category" functionality in developer settings, with "All" as a default option. By combining data categorization with the prepdocs.py scripts, it lets you filter Azure AI Search based on categories. This functionality allows category-based inquiries and informs the user about the categories present in the search index.

This PR is somehow related to the following past issues:

• Switching between search index #2006
• How to define/exclude categories? #1715
• Improve usability of categories - make it multiselect control #1333
• Frontend support to switch between indexes #872
• Documentation please: Exclude category #647
• Feature request: Respond with whether the model knows about a file #493

Changes

• Modified approach.py and models.ts to include category logic
• Added translations for category ("All") in en, es, fr, and ja locales
• Updated Ask.tsx and Chat.tsx to handle category
• Updated data ingestion documentation

Quick insight

Here is an example of how categories appear when ingested into the search index:

1. "category": "BenefitOptions"
2. "category": "RoleLibrary"
3. "category": "EmployeeHandbook"

Let's add category keys like this:

Then, in the default "All" category, we get answers from all three categories, like this:

When specifying a particular category, we get generated answers from that category's data:

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[ ] Yes
[x] No

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[x] Yes
[ ] No

Type of change

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

• The current tests all pass (python -m pytest).
• I added tests that prove my fix is effective or that my feature works
• I ran python -m pytest --cov to verify 100% coverage of added lines
• I ran python -m mypy to check for type errors
• I either used the pre-commit hooks or ran ruff and black manually on my code.

[jeannot-prvlimburg]

Are there any updates on this feature?

[pamelafox]

@glaucia86 Could you suggest the pt-br strings for "All" and "Include category" so @bnodir can add to the new pt-br file for this PR?

[glaucia86]

For Spanish, sounds pretty good for me

[pamelafox]

We need pt-br strings since I just merged that file, and I've asked humans to review the other strings.

[pamelafox]

@glaucia86 Thank you! Can you also provide the portuguese translations of those strings? The pt-br file isn't showing up in this PR since I just merged it in, but it needs updating with the same new strings.

[bnodir]

Thank you, everyone. I have made the changes based on the reviewers' feedback.

[zedhaque]

Just wondering, if the category isn't mandatory and some documents don't have a category, will the "All" option include them? Thanks

[bnodir]

@zedhaque - Yes, the "All" option includes documents that don't have a category as well.

[pamelafox]

I think we may have some confusion with "All", with people wondering if that includes uncategorized, but we can watch and see if any issues come in about that. Thank you! I'll ask @mattgotteiner for a review as well since this adds a search filter.

[mattgotteiner]

Thank you for your contribution!

[cforce]

Does "All" also includes Uploaded Documents? It would be great to have as well "Uploaded" or "MyDocs"

[jeannot-prvlimburg]

Does "All" also includes Uploaded Documents? It would be great to have as well "Uploaded" or "MyDocs"

Yes, it does! It would indeed be helpful to tag manually uploaded documents with an "Uploaded" or "MyDocs" category to distinguish them easily. Additionally, supporting multiple categories would enable users to filter and select specific document subsets more precisely.

[bnodir]

@jeannot-prvlimburg

I extended the list of choices in app/frontend/src/locales/en/translation.json to include:
"includeCategoryOptions": { "all": "All", "digitalisering": "Digitalisering" }
However, only "All" appears as an option in the dropdown, even though I'm using English as the language. Could you advise on what might be missing for the dropdown to display the additional option(s)?

Ensure your code looks like this:

                        aria-labelledby={includeCategoryId}
                        options={[
                            { key: "", text: t("labels.includeCategoryOptions.all") },   // ← Add comma
                            { key: "digitalisering", text: "Digitalisering" }   // ← Added new category line
                            // You can add a category key here for ingested data like below:
                            // { key: 'categoryName', text: 'Meaningful Category Name' }
                            // Alternatively, display the key to guide the user on what to type
                            // in the "Exclude category" field (e.g., 'Meaningful Category Name(categoryName)').
                        ]}

This should display the additional option in the dropdown. Additionally, please ensure to refresh your browser to see the changes.

[jeannot-prvlimburg]

@jeannot-prvlimburg

I extended the list of choices in app/frontend/src/locales/en/translation.json to include:
"includeCategoryOptions": { "all": "All", "digitalisering": "Digitalisering" }
However, only "All" appears as an option in the dropdown, even though I'm using English as the language. Could you advise on what might be missing for the dropdown to display the additional option(s)?

Ensure your code looks like this:

                        aria-labelledby={includeCategoryId}
                        options={[
                            { key: "", text: t("labels.includeCategoryOptions.all") },   // ← Add comma
                            { key: "digitalisering", text: "Digitalisering" }   // ← Added new category line
                            // You can add a category key here for ingested data like below:
                            // { key: 'categoryName', text: 'Meaningful Category Name' }
                            // Alternatively, display the key to guide the user on what to type
                            // in the "Exclude category" field (e.g., 'Meaningful Category Name(categoryName)').
                        ]}

This should display the additional option in the dropdown. Additionally, please ensure to refresh your browser to see the changes.

I found out indeed. Thanks for the help!