MultimodalQnA Image and Audio Support Phase 1 by mhbuehler · Pull Request #1071 · opea-project/GenAIExamples

mhbuehler · 2024-11-05T00:52:13Z

Description

This PR adds the following new features as specified in "Phase 1" of this RFC. The related PR in GenAIComps is opea-project/GenAIComps#852 and it will need to be merged before this one in GenAIExamples.

Data prep and ingestion enhancements:

Accept image only
Accept image and text
Accept speech audio only

Other enhancements:

Allow the user to choose the embedding model and LVM when starting the services

Note that the planned query enhancement "Accept speech audio only" has been moved to Phase 2 and a PR for that phase will be submitted for the next release.

Issues

MultimodalQnA Image & Audio Support RFC

Type of change

List the type of change like below. Please delete options that are not relevant.

New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

No new dependencies

Tests

Updated the individual microservice's test scripts, the GenAIExamples' MultimodalQnA test scripts, and did manual testing of the UI and documented curl commands.

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* added entrypoint override Signed-off-by: okhleif-IL <omar.khleif@intel.com> * fixed entrypoint container Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed workdir Signed-off-by: okhleif-IL <omar.khleif@intel.com> * reverted set env vars changes Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added entrypoint for embedding model Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added llava model to readme Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed stray \n Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added entrypoint override in bridgetower hpu Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…e/mm-rag-enhanced

* changed variable names Signed-off-by: okhleif-IL <omar.khleif@intel.com> * changed gaudi env vars Signed-off-by: okhleif-IL <omar.khleif@intel.com> * changed vars + docs Signed-off-by: okhleif-IL <omar.khleif@intel.com> * verified + updated readme Signed-off-by: okhleif-IL <omar.khleif@intel.com> * updated readme Signed-off-by: okhleif-IL <omar.khleif@intel.com> * updated tests Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…estion Image ingestion improvements

…e/mm-rag-enhanced

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Update LVM model for Xeon

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

…e/mm-rag-enhanced

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Image+Text Upload

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

UI Audio Upload

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

UI fixes

Updated main README and supported_examples with MMQA details

for more information, see https://pre-commit.ci

Fix for lvm-dependent tests and clear textbox

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

MultimodalQnA/tests/test_compose_on_gaudi.sh

MultimodalQnA/tests/test_compose_on_xeon.sh

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Fixed typos for MMQA in Docker Image List doc

for more information, see https://pre-commit.ci

Clone specific branch of GenAIComps for tests

ashahba

Putting this PR on hold until opea-project/GenAIComps#852 is merged.

MultimodalQnA/tests/test_compose_on_gaudi.sh

MultimodalQnA/tests/test_compose_on_xeon.sh

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

for more information, see https://pre-commit.ci

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Test update: Increase sleep to 3m to wait for LVM microservice to be ready

Spycsh · 2024-11-08T02:23:30Z

opea-project/GenAIComps#852 is merged now, please resolve comments from @ashahba .

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Revert change of repo for tests

ashahba

LGTM!

ashahba · 2024-11-08T06:22:03Z

@tileintel this one is ready to be merged.
All I need is your 👍
Thanks.

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Signed-off-by: okhleif-IL <omar.khleif@intel.com> Signed-off-by: dmsuehir <dina.s.jones@intel.com> Co-authored-by: Omar Khleif <omar.khleif@intel.com> Co-authored-by: dmsuehir <dina.s.jones@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com> Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>

mhbuehler and others added 30 commits October 14, 2024 16:28

Initial implementation of image ingestion

97f4bc0

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Use same endpoint for image and video ingestion

56bd8c3

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Update tests and docs

b334dc2

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

13c752a

…e/mm-rag-enhanced

added LVM_MODEL_ID var to test file (#4)

271117e

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Updates tests per feedback

c4e0259

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge pull request #2 from mhbuehler/melanie/combined_image_video_ing…

a541140

…estion Image ingestion improvements

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

30c311d

…e/mm-rag-enhanced

Update LVM model for Xeon

e83fc44

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Merge pull request #5 from mhbuehler/dina/lvm_model

4b8a5ad

Update LVM model for Xeon

Initial setup for ingest_with_text

a6d826c

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Write and send custom caption file

547a139

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Update docs and tests

7cfc343

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

MMQnA doc updates for audio ingestion (#7)

69dbdfc

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

834c668

…e/mm-rag-enhanced

Fix UI request for follow up queries with no image (#8)

a8f8dc9

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Updated for review suggestions

718d02e

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge branch 'melanie/mm-rag-enhanced' into melanie/images_and_text

d05cfb3

Add audio upload functionality to UI

72591c1

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge pull request #6 from mhbuehler/melanie/images_and_text

431e41b

Image+Text Upload

Merge branch 'melanie/mm-rag-enhanced' into melanie/audio_ingest_ui

d535aa7

Minor refactor, improve display text, and suppress PDF tab

39f43fc

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge pull request #9 from mhbuehler/melanie/audio_ingest_ui

afc3c8a

UI Audio Upload

Small fixes

ddd5dfb

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Improve appearance

426c739

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Improve upload errors and revert multimodal query box

cfa1c8c

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Small text edit as suggested

cdec83f

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge pull request #11 from mhbuehler/melanie/mm-fixes

48baceb

UI fixes

mhbuehler and others added 4 commits November 6, 2024 09:30

Merge pull request #13 from mhbuehler/omar/mmqa-docs

ee387a2

Updated main README and supported_examples with MMQA details

[pre-commit.ci] auto fixes from pre-commit.com hooks

aafcfe1

for more information, see https://pre-commit.ci

Merge pull request #14 from mhbuehler/melanie/fix_tests

359b6f8

Fix for lvm-dependent tests and clear textbox

fixed multimodalqna typos

9d3ed45

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

ashahba reviewed Nov 6, 2024

View reviewed changes

MultimodalQnA/tests/test_compose_on_gaudi.sh Show resolved Hide resolved

ashahba reviewed Nov 6, 2024

View reviewed changes

MultimodalQnA/tests/test_compose_on_xeon.sh Show resolved Hide resolved

mhbuehler and others added 4 commits November 6, 2024 10:30

Point git clone at specific branch of GenAIComps

54cff40

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge pull request #15 from mhbuehler/omar/dockimg_doc

fd9fd84

Fixed typos for MMQA in Docker Image List doc

[pre-commit.ci] auto fixes from pre-commit.com hooks

d88513a

for more information, see https://pre-commit.ci

Merge pull request #16 from mhbuehler/melanie/clone_specific_branch

24438e5

Clone specific branch of GenAIComps for tests

ashahba requested changes Nov 6, 2024

View reviewed changes

MultimodalQnA/tests/test_compose_on_gaudi.sh Outdated Show resolved Hide resolved

MultimodalQnA/tests/test_compose_on_xeon.sh Outdated Show resolved Hide resolved

dmsuehir and others added 7 commits November 6, 2024 14:34

Fix xeon test to use llava-hf/llava-1.5-7b-hf (#17)

df2511b

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Update MMQnA xeon test to wait for LVM to be ready (#18)

6631601

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

59acc77

for more information, see https://pre-commit.ci

Test update: Increase wait time and add more messages (#19)

9d4cb5f

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Test update: Increase sleep

c1bfe6a

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Merge pull request #20 from mhbuehler/dina/test-update-wait

b88aa3f

Test update: Increase sleep to 3m to wait for LVM microservice to be ready

Merge branch 'main' into melanie/mm-rag-enhanced

6e5a200

lvliang-intel requested a review from Spycsh November 8, 2024 01:53

Spycsh approved these changes Nov 8, 2024

View reviewed changes

mhbuehler and others added 3 commits November 7, 2024 21:52

Revert change of repo for tests

e2c585e

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Merge pull request #21 from mhbuehler/melanie/revert_repo_change

45c7cf9

Revert change of repo for tests

Merge branch 'main' into melanie/mm-rag-enhanced

04fede6

ashahba approved these changes Nov 8, 2024

View reviewed changes

joshuayao linked an issue Nov 8, 2024 that may be closed by this pull request

Image and Audio Support for MultimodalityQnA opea-project/GenAIComps#790

Closed

ftian1 merged commit bbc95bb into opea-project:main Nov 8, 2024

dmsuehir deleted the melanie/mm-rag-enhanced branch March 19, 2025 23:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultimodalQnA Image and Audio Support Phase 1#1071

MultimodalQnA Image and Audio Support Phase 1#1071
ftian1 merged 58 commits intoopea-project:mainfrom
mhbuehler:melanie/mm-rag-enhanced

mhbuehler commented Nov 5, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ashahba left a comment

Uh oh!

Uh oh!

Uh oh!

Spycsh commented Nov 8, 2024

Uh oh!

ashahba left a comment

Uh oh!

ashahba commented Nov 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

mhbuehler commented Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

Uh oh!

Uh oh!

ashahba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Spycsh commented Nov 8, 2024

Uh oh!

ashahba left a comment

Choose a reason for hiding this comment

Uh oh!

ashahba commented Nov 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mhbuehler commented Nov 5, 2024 •

edited

Loading