Add image inference support #8954

gsiddh · 2025-04-22T03:19:02Z

Adding image inference support.

changeset-bot · 2025-04-22T03:19:05Z

⚠️ No Changeset found

Latest commit: 8cdb35c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

github-actions · 2025-04-22T03:19:17Z

Vertex AI Mock Responses Check ⚠️

A newer major version of the mock responses for Vertex AI unit tests is available. update_vertexai_responses.sh should be updated to clone the latest version of the responses: v10.0

google-oss-bot · 2025-04-22T03:28:43Z

Size Report ¹

Affected Products

@firebase/vertexai
Type Base (a46fa4a) Merge (d2a41fc) Diff
browser 38.9 kB 39.5 kB +566 B (+1.5%)
main 39.9 kB 40.5 kB +566 B (+1.4%)
module 38.9 kB 39.5 kB +566 B (+1.5%)
firebase
Type Base (a46fa4a) Merge (d2a41fc) Diff
firebase-vertexai.js 31.2 kB 31.6 kB +429 B (+1.4%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/qyhWYZTP7I.html

google-oss-bot · 2025-04-22T03:40:03Z

Size Analysis Report ¹

Affected Products

@firebase/vertexai
- getGenerativeModel
  Size
  Type Base (a46fa4a) Merge (d2a41fc) Diff
  size 22.0 kB 22.4 kB +417 B (+1.9%)
  size-with-ext-deps 41.0 kB 41.4 kB +417 B (+1.0%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/clm2afjJgF.html

erikeldridge · 2025-04-22T03:45:31Z

packages/vertexai/src/methods/chrome-adapter.test.ts

+      const promptOutput = 'hi';
+      const promptStub = stub(languageModel, 'prompt').resolves(promptOutput);
+      const onDeviceParams = {
+        systemPrompt: 'be yourself'


I think a more relevant test would be to assert onDeviceParams.expectedInputs is threaded through, since that's required for image input to work on-device.

oh yes. thanks for suggesting that. I was thinking to add it in - totally forgot about it. Adding it now.

solving in next change within this PR.

packages/vertexai/src/methods/chrome-adapter.ts

packages/vertexai/src/methods/chrome-adapter.test.ts

packages/vertexai/src/methods/chrome-adapter.ts

erikeldridge

Shipit! 🚀

* Adding image based input for inference * adding image as input to create language model object

Fix languageCode parameter in action_code_url (#8912) * Fix languageCode parameter in action_code_url * Add changeset Vaihi add langmodel types. (#8927) * Adding LanguageModel types. These are based off https://github.com/webmachinelearning/prompt-api?tab=readme-ov-file#full-api-surface-in-web-idl * Adding LanguageModel types. * Remove bunch of exports * yarn formatted * after lint Define HybridParams (#8935) Co-authored-by: Erik Eldridge <erikeldridge@google.com> Adding smoke test for new hybrid params (#8937) * Adding smoke test for new hybrid params * Use the existing name of the model params input --------- Co-authored-by: Erik Eldridge <erikeldridge@google.com> Moving to in-cloud naming (#8938) Co-authored-by: Erik Eldridge <erikeldridge@google.com> Moving to string type for the inference mode (#8941) Define ChromeAdapter class (#8942) Co-authored-by: Erik Eldridge <erikeldridge@google.com> VinF Hybrid Inference: Implement ChromeAdapter (rebased) (#8943) Adding count token impl (#8950) VinF Hybrid Inference #4: ChromeAdapter in stream methods (rebased) (#8949) Define values for Availability enum (#8951) VinF Hybrid Inference: narrow Chrome input type (#8953) Add image inference support (#8954) * Adding image based input for inference * adding image as input to create language model object disable count tokens api for on-device inference (#8962) VinF Hybrid Inference: throw if only_on_device and model is unavailable (#8965)

gsiddh requested a review from a team as a code owner April 22, 2025 03:19

gsiddh force-pushed the vaihi-image-inf branch 3 times, most recently from 734a1ca to f815c59 Compare April 22, 2025 03:53

gsiddh changed the title ~~first pass~~ Add image inference support Apr 22, 2025

erikeldridge approved these changes Apr 22, 2025

View reviewed changes

erikeldridge reviewed Apr 22, 2025

View reviewed changes

packages/vertexai/src/methods/chrome-adapter.ts Outdated Show resolved Hide resolved

Adding image based input for inference

60b50a1

gsiddh force-pushed the vaihi-image-inf branch from f815c59 to 60b50a1 Compare April 22, 2025 04:49

adding image as input to create language model object

8cdb35c

erikeldridge approved these changes Apr 22, 2025

View reviewed changes

gsiddh merged commit 43a69d5 into vaihi-exp Apr 22, 2025
40 of 46 checks passed

gsiddh deleted the vaihi-image-inf branch April 22, 2025 15:49

gsiddh added a commit that referenced this pull request Apr 22, 2025

Add image inference support (#8954)

76ef394

* Adding image based input for inference * adding image as input to create language model object

gsiddh added a commit that referenced this pull request Apr 22, 2025

Add image inference support (#8954)

2a503a9

* Adding image based input for inference * adding image as input to create language model object

gsiddh mentioned this pull request Apr 23, 2025

Complete Hybrid inference impl #8966

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add image inference support #8954

Add image inference support #8954

gsiddh commented Apr 22, 2025 •

edited

Loading

changeset-bot bot commented Apr 22, 2025 •

edited

Loading

github-actions bot commented Apr 22, 2025

google-oss-bot commented Apr 22, 2025 •

edited

Loading

`@firebase/vertexai`

`firebase`

google-oss-bot commented Apr 22, 2025 •

edited

Loading

`@firebase/vertexai`

`getGenerativeModel`

erikeldridge Apr 22, 2025

gsiddh Apr 22, 2025

gsiddh Apr 22, 2025

erikeldridge left a comment

Add image inference support #8954

Add image inference support #8954

Conversation

gsiddh commented Apr 22, 2025 • edited Loading

changeset-bot bot commented Apr 22, 2025 • edited Loading

⚠️ No Changeset found

github-actions bot commented Apr 22, 2025

Vertex AI Mock Responses Check ⚠️

google-oss-bot commented Apr 22, 2025 • edited Loading

Size Report 1

Affected Products

@firebase/vertexai

firebase

Test Logs

google-oss-bot commented Apr 22, 2025 • edited Loading

Size Analysis Report 1

Affected Products

@firebase/vertexai

getGenerativeModel

Test Logs

erikeldridge Apr 22, 2025

Choose a reason for hiding this comment

gsiddh Apr 22, 2025

Choose a reason for hiding this comment

gsiddh Apr 22, 2025

Choose a reason for hiding this comment

erikeldridge left a comment

Choose a reason for hiding this comment

gsiddh commented Apr 22, 2025 •

edited

Loading

changeset-bot bot commented Apr 22, 2025 •

edited

Loading

google-oss-bot commented Apr 22, 2025 •

edited

Loading

Size Report ¹

`@firebase/vertexai`

`firebase`

google-oss-bot commented Apr 22, 2025 •

edited

Loading

Size Analysis Report ¹

`@firebase/vertexai`

`getGenerativeModel`