Skip to content

Commit ce2c775

Browse files
dlarocquedlarocque
authored
Add Image Generation support using Imagen (#8683)
* WIP * Introduce VertexAIModel base class, add documentation, and respond to other comments * Generate devsite docs * revert dataconnect change * formatting * WIP * Introduce VertexAIModel base class, add documentation, and respond to other comments * Generate devsite docs * revert dataconnect change * formatting * Fix ImagenRequestConfig comment * Improve tests * Formatting * Add ImagenGenerationResponse * Formatting * Update to match API changes * fixes * format * Update API reports * add changeset * update * format * Update API reports * remove public docs * Nest imageFormat in outputOptions * remove unecessary console.log * Log warning if compressionQuality outside of range * format * Imagen Documentation (#8776) * Revert "remove public docs" This reverts commit f095563. * update docs * update docs * format * minor fixes * format * mark all APIs beta instead of public * review fixes * format * Replace (BETA) with (Public Preview) in docs * fix links * another pass * format * review fixes * Cleanup * update changeset --------- Co-authored-by: dlarocque <dlarocque@users.noreply.github.com>
1 parent c3266f8 commit ce2c775

35 files changed

+2427
-103
lines changed

‎.changeset/violet-planets-impress.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'firebase': minor
3+
'@firebase/vertexai': minor
4+
---
5+
6+
**Public Preview** Added support for generating images using the Imagen 3 model.

‎common/api-review/vertexai.api.md‎

Lines changed: 98 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -323,16 +323,14 @@ export interface GenerativeContentBlob {
323323
}
324324

325325
// @public
326-
export class GenerativeModel {
326+
export class GenerativeModel extends VertexAIModel {
327327
constructor(vertexAI: VertexAI, modelParams: ModelParams, requestOptions?: RequestOptions);
328328
countTokens(request: CountTokensRequest | string | Array<string | Part>): Promise<CountTokensResponse>;
329329
generateContent(request: GenerateContentRequest | string | Array<string | Part>): Promise<GenerateContentResult>;
330330
generateContentStream(request: GenerateContentRequest | string | Array<string | Part>): Promise<GenerateContentStreamResult>;
331331
// (undocumented)
332332
generationConfig: GenerationConfig;
333333
// (undocumented)
334-
model: string;
335-
// (undocumented)
336334
requestOptions?: RequestOptions;
337335
// (undocumented)
338336
safetySettings: SafetySetting[];
@@ -348,6 +346,9 @@ export class GenerativeModel {
348346
// @public
349347
export function getGenerativeModel(vertexAI: VertexAI, modelParams: ModelParams, requestOptions?: RequestOptions): GenerativeModel;
350348

349+
// @beta
350+
export function getImagenModel(vertexAI: VertexAI, modelParams: ImagenModelParams, requestOptions?: RequestOptions): ImagenModel;
351+
351352
// @public
352353
export function getVertexAI(app?: FirebaseApp, options?: VertexAIOptions): VertexAI;
353354

@@ -429,6 +430,90 @@ export enum HarmSeverity {
429430
HARM_SEVERITY_NEGLIGIBLE = "HARM_SEVERITY_NEGLIGIBLE"
430431
}
431432

433+
// @beta
434+
export enum ImagenAspectRatio {
435+
LANDSCAPE_16x9 = "16:9",
436+
LANDSCAPE_3x4 = "3:4",
437+
PORTRAIT_4x3 = "4:3",
438+
PORTRAIT_9x16 = "9:16",
439+
SQUARE = "1:1"
440+
}
441+
442+
// @public
443+
export interface ImagenGCSImage {
444+
gcsURI: string;
445+
mimeType: string;
446+
}
447+
448+
// @beta
449+
export interface ImagenGenerationConfig {
450+
addWatermark?: boolean;
451+
aspectRatio?: ImagenAspectRatio;
452+
imageFormat?: ImagenImageFormat;
453+
negativePrompt?: string;
454+
numberOfImages?: number;
455+
}
456+
457+
// @beta
458+
export interface ImagenGenerationResponse<T extends ImagenInlineImage | ImagenGCSImage> {
459+
filteredReason?: string;
460+
images: T[];
461+
}
462+
463+
// @beta
464+
export class ImagenImageFormat {
465+
compressionQuality?: number;
466+
static jpeg(compressionQuality?: number): ImagenImageFormat;
467+
mimeType: string;
468+
static png(): ImagenImageFormat;
469+
}
470+
471+
// @beta
472+
export interface ImagenInlineImage {
473+
bytesBase64Encoded: string;
474+
mimeType: string;
475+
}
476+
477+
// @beta
478+
export class ImagenModel extends VertexAIModel {
479+
constructor(vertexAI: VertexAI, modelParams: ImagenModelParams, requestOptions?: RequestOptions | undefined);
480+
generateImages(prompt: string): Promise<ImagenGenerationResponse<ImagenInlineImage>>;
481+
// @internal
482+
generateImagesGCS(prompt: string, gcsURI: string): Promise<ImagenGenerationResponse<ImagenGCSImage>>;
483+
generationConfig?: ImagenGenerationConfig;
484+
// (undocumented)
485+
requestOptions?: RequestOptions | undefined;
486+
safetySettings?: ImagenSafetySettings;
487+
}
488+
489+
// @beta
490+
export interface ImagenModelParams {
491+
generationConfig?: ImagenGenerationConfig;
492+
model: string;
493+
safetySettings?: ImagenSafetySettings;
494+
}
495+
496+
// @beta
497+
export enum ImagenPersonFilterLevel {
498+
ALLOW_ADULT = "allow_adult",
499+
ALLOW_ALL = "allow_all",
500+
BLOCK_ALL = "dont_allow"
501+
}
502+
503+
// @beta
504+
export enum ImagenSafetyFilterLevel {
505+
BLOCK_LOW_AND_ABOVE = "block_low_and_above",
506+
BLOCK_MEDIUM_AND_ABOVE = "block_medium_and_above",
507+
BLOCK_NONE = "block_none",
508+
BLOCK_ONLY_HIGH = "block_only_high"
509+
}
510+
511+
// @beta
512+
export interface ImagenSafetySettings {
513+
personFilterLevel?: ImagenPersonFilterLevel;
514+
safetyFilterLevel?: ImagenSafetyFilterLevel;
515+
}
516+
432517
// @public
433518
export interface InlineDataPart {
434519
// (undocumented)
@@ -718,6 +803,16 @@ export const enum VertexAIErrorCode {
718803
RESPONSE_ERROR = "response-error"
719804
}
720805

806+
// @public
807+
export abstract class VertexAIModel {
808+
// @internal
809+
protected constructor(vertexAI: VertexAI, modelName: string);
810+
// @internal (undocumented)
811+
protected _apiSettings: ApiSettings;
812+
readonly model: string;
813+
static normalizeModelName(modelName: string): string;
814+
}
815+
721816
// @public
722817
export interface VertexAIOptions {
723818
// (undocumented)

‎docs-devsite/_toc.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -536,6 +536,22 @@ toc:
536536
path: /docs/reference/js/vertexai.groundingattribution.md
537537
- title: GroundingMetadata
538538
path: /docs/reference/js/vertexai.groundingmetadata.md
539+
- title: ImagenGCSImage
540+
path: /docs/reference/js/vertexai.imagengcsimage.md
541+
- title: ImagenGenerationConfig
542+
path: /docs/reference/js/vertexai.imagengenerationconfig.md
543+
- title: ImagenGenerationResponse
544+
path: /docs/reference/js/vertexai.imagengenerationresponse.md
545+
- title: ImagenImageFormat
546+
path: /docs/reference/js/vertexai.imagenimageformat.md
547+
- title: ImagenInlineImage
548+
path: /docs/reference/js/vertexai.imageninlineimage.md
549+
- title: ImagenModel
550+
path: /docs/reference/js/vertexai.imagenmodel.md
551+
- title: ImagenModelParams
552+
path: /docs/reference/js/vertexai.imagenmodelparams.md
553+
- title: ImagenSafetySettings
554+
path: /docs/reference/js/vertexai.imagensafetysettings.md
539555
- title: InlineDataPart
540556
path: /docs/reference/js/vertexai.inlinedatapart.md
541557
- title: IntegerSchema
@@ -584,6 +600,8 @@ toc:
584600
path: /docs/reference/js/vertexai.vertexai.md
585601
- title: VertexAIError
586602
path: /docs/reference/js/vertexai.vertexaierror.md
603+
- title: VertexAIModel
604+
path: /docs/reference/js/vertexai.vertexaimodel.md
587605
- title: VertexAIOptions
588606
path: /docs/reference/js/vertexai.vertexaioptions.md
589607
- title: VideoMetadata

‎docs-devsite/vertexai.generativemodel.md‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,9 @@ Class for generative model APIs.
1515
<b>Signature:</b>
1616

1717
```typescript
18-
export declare class GenerativeModel
18+
export declare class GenerativeModel extends VertexAIModel
1919
```
20+
<b>Extends:</b> [VertexAIModel](./vertexai.vertexaimodel.md#vertexaimodel_class)
2021
2122
## Constructors
2223
@@ -29,7 +30,6 @@ export declare class GenerativeModel
2930
| Property | Modifiers | Type | Description |
3031
| --- | --- | --- | --- |
3132
| [generationConfig](./vertexai.generativemodel.md#generativemodelgenerationconfig) | | [GenerationConfig](./vertexai.generationconfig.md#generationconfig_interface) | |
32-
| [model](./vertexai.generativemodel.md#generativemodelmodel) | | string | |
3333
| [requestOptions](./vertexai.generativemodel.md#generativemodelrequestoptions) | | [RequestOptions](./vertexai.requestoptions.md#requestoptions_interface) | |
3434
| [safetySettings](./vertexai.generativemodel.md#generativemodelsafetysettings) | | [SafetySetting](./vertexai.safetysetting.md#safetysetting_interface)<!-- -->\[\] | |
3535
| [systemInstruction](./vertexai.generativemodel.md#generativemodelsysteminstruction) | | [Content](./vertexai.content.md#content_interface) | |
@@ -71,14 +71,6 @@ constructor(vertexAI: VertexAI, modelParams: ModelParams, requestOptions?: Reque
7171
generationConfig: GenerationConfig;
7272
```
7373
74-
## GenerativeModel.model
75-
76-
<b>Signature:</b>
77-
78-
```typescript
79-
model: string;
80-
```
81-
8274
## GenerativeModel.requestOptions
8375
8476
<b>Signature:</b>

‎docs-devsite/vertexai.imagengcsimage.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
Project: /docs/reference/js/_project.yaml
2+
Book: /docs/reference/_book.yaml
3+
page_type: reference
4+
5+
{% comment %}
6+
DO NOT EDIT THIS FILE!
7+
This is generated by the JS SDK team, and any local changes will be
8+
overwritten. Changes should be made in the source code at
9+
https://github.com/firebase/firebase-js-sdk
10+
{% endcomment %}
11+
12+
# ImagenGCSImage interface
13+
An image generated by Imagen, stored in a Cloud Storage for Firebase bucket.
14+
15+
This feature is not available yet.
16+
17+
<b>Signature:</b>
18+
19+
```typescript
20+
export interface ImagenGCSImage
21+
```
22+
23+
## Properties
24+
25+
| Property | Type | Description |
26+
| --- | --- | --- |
27+
| [gcsURI](./vertexai.imagengcsimage.md#imagengcsimagegcsuri) | string | The URI of the file stored in a Cloud Storage for Firebase bucket. |
28+
| [mimeType](./vertexai.imagengcsimage.md#imagengcsimagemimetype) | string | The MIME type of the image; either <code>&quot;image/png&quot;</code> or <code>&quot;image/jpeg&quot;</code>.<!-- -->To request a different format, set the <code>imageFormat</code> property in your <code>[ImagenGenerationConfig](./vertexai.imagengenerationconfig.md#imagengenerationconfig_interface)</code>. |
29+
30+
## ImagenGCSImage.gcsURI
31+
32+
The URI of the file stored in a Cloud Storage for Firebase bucket.
33+
34+
<b>Signature:</b>
35+
36+
```typescript
37+
gcsURI: string;
38+
```
39+
40+
### Example
41+
42+
`"gs://bucket-name/path/sample_0.jpg"`<!-- -->.
43+
44+
## ImagenGCSImage.mimeType
45+
46+
The MIME type of the image; either `"image/png"` or `"image/jpeg"`<!-- -->.
47+
48+
To request a different format, set the `imageFormat` property in your <code>[ImagenGenerationConfig](./vertexai.imagengenerationconfig.md#imagengenerationconfig_interface)</code>.
49+
50+
<b>Signature:</b>
51+
52+
```typescript
53+
mimeType: string;
54+
```

‎docs-devsite/vertexai.imagengenerationconfig.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
Project: /docs/reference/js/_project.yaml
2+
Book: /docs/reference/_book.yaml
3+
page_type: reference
4+
5+
{% comment %}
6+
DO NOT EDIT THIS FILE!
7+
This is generated by the JS SDK team, and any local changes will be
8+
overwritten. Changes should be made in the source code at
9+
https://github.com/firebase/firebase-js-sdk
10+
{% endcomment %}
11+
12+
# ImagenGenerationConfig interface
13+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
14+
>
15+
16+
Configuration options for generating images with Imagen.
17+
18+
See the [documentation](http://firebase.google.com/docs/vertex-ai/generate-images-imagen) for more details.
19+
20+
<b>Signature:</b>
21+
22+
```typescript
23+
export interface ImagenGenerationConfig
24+
```
25+
26+
## Properties
27+
28+
| Property | Type | Description |
29+
| --- | --- | --- |
30+
| [addWatermark](./vertexai.imagengenerationconfig.md#imagengenerationconfigaddwatermark) | boolean | <b><i>(Public Preview)</i></b> Whether to add an invisible watermark to generated images.<!-- -->If set to <code>true</code>, an invisible SynthID watermark is embedded in generated images to indicate that they are AI generated. If set to <code>false</code>, watermarking will be disabled.<!-- -->For Imagen 3 models, the default value is <code>true</code>; see the <a href="http://firebase.google.com/docs/vertex-ai/model-parameters#imagen"><code>addWatermark</code></a> documentation for more details. |
31+
| [aspectRatio](./vertexai.imagengenerationconfig.md#imagengenerationconfigaspectratio) | [ImagenAspectRatio](./vertexai.md#imagenaspectratio) | <b><i>(Public Preview)</i></b> The aspect ratio of the generated images. The default value is square 1:1. Supported aspect ratios depend on the Imagen model, see <code>[ImagenAspectRatio](./vertexai.md#imagenaspectratio)</code> for more details. |
32+
| [imageFormat](./vertexai.imagengenerationconfig.md#imagengenerationconfigimageformat) | [ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class) | <b><i>(Public Preview)</i></b> The image format of the generated images. The default is PNG.<!-- -->See <code>[ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class)</code> for more details. |
33+
| [negativePrompt](./vertexai.imagengenerationconfig.md#imagengenerationconfignegativeprompt) | string | <b><i>(Public Preview)</i></b> A description of what should be omitted from the generated images.<!-- -->Support for negative prompts depends on the Imagen model.<!-- -->See the [documentation](http://firebase.google.com/docs/vertex-ai/model-parameters#imagen) for more details. |
34+
| [numberOfImages](./vertexai.imagengenerationconfig.md#imagengenerationconfignumberofimages) | number | <b><i>(Public Preview)</i></b> The number of images to generate. The default value is 1.<!-- -->The number of sample images that may be generated in each request depends on the model (typically up to 4); see the <a href="http://firebase.google.com/docs/vertex-ai/model-parameters#imagen"><code>sampleCount</code></a> documentation for more details. |
35+
36+
## ImagenGenerationConfig.addWatermark
37+
38+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
39+
>
40+
41+
Whether to add an invisible watermark to generated images.
42+
43+
If set to `true`<!-- -->, an invisible SynthID watermark is embedded in generated images to indicate that they are AI generated. If set to `false`<!-- -->, watermarking will be disabled.
44+
45+
For Imagen 3 models, the default value is `true`<!-- -->; see the <a href="http://firebase.google.com/docs/vertex-ai/model-parameters#imagen"><code>addWatermark</code></a> documentation for more details.
46+
47+
<b>Signature:</b>
48+
49+
```typescript
50+
addWatermark?: boolean;
51+
```
52+
53+
## ImagenGenerationConfig.aspectRatio
54+
55+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
56+
>
57+
58+
The aspect ratio of the generated images. The default value is square 1:1. Supported aspect ratios depend on the Imagen model, see <code>[ImagenAspectRatio](./vertexai.md#imagenaspectratio)</code> for more details.
59+
60+
<b>Signature:</b>
61+
62+
```typescript
63+
aspectRatio?: ImagenAspectRatio;
64+
```
65+
66+
## ImagenGenerationConfig.imageFormat
67+
68+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
69+
>
70+
71+
The image format of the generated images. The default is PNG.
72+
73+
See <code>[ImagenImageFormat](./vertexai.imagenimageformat.md#imagenimageformat_class)</code> for more details.
74+
75+
<b>Signature:</b>
76+
77+
```typescript
78+
imageFormat?: ImagenImageFormat;
79+
```
80+
81+
## ImagenGenerationConfig.negativePrompt
82+
83+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
84+
>
85+
86+
A description of what should be omitted from the generated images.
87+
88+
Support for negative prompts depends on the Imagen model.
89+
90+
See the [documentation](http://firebase.google.com/docs/vertex-ai/model-parameters#imagen) for more details.
91+
92+
<b>Signature:</b>
93+
94+
```typescript
95+
negativePrompt?: string;
96+
```
97+
98+
## ImagenGenerationConfig.numberOfImages
99+
100+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
101+
>
102+
103+
The number of images to generate. The default value is 1.
104+
105+
The number of sample images that may be generated in each request depends on the model (typically up to 4); see the <a href="http://firebase.google.com/docs/vertex-ai/model-parameters#imagen"><code>sampleCount</code></a> documentation for more details.
106+
107+
<b>Signature:</b>
108+
109+
```typescript
110+
numberOfImages?: number;
111+
```

0 commit comments

Comments
 (0)