Skip to content

feat(component,ai,gemini): implement File API support for large files… #1118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 22, 2025
Merged

feat(component,ai,gemini): implement File API support for large files… #1118

merged 1 commit into from
Sep 22, 2025

Conversation

@pinglin
Copy link
Member

@pinglin pinglin commented Sep 22, 2025

Because

  • The Gemini API has a 20MB limit for inline data in requests, which might cause failures when processing large files
  • Video files for caching operations require File API usage regardless of size due to Gemini API requirements
  • Large multimedia files (images, audio, documents) above 20MB could not be processed effectively
  • The existing implementation only supported inline base64 encoding, limiting file size capabilities

This commit

  • Implements Gemini File API integration: Adds support for uploading large files (>20MB) and all video files to Gemini's File API before processing
  • Adds intelligent size-based routing: Files are automatically processed via File API when total request size exceeds 20MB, with special handling for cache video files
  • Introduces file lifecycle management: Uploaded files are automatically deleted from File API after processing to prevent storage accumulation
  • Adds file state polling: Implements robust waiting mechanism for uploaded files to reach ACTIVE state before use
  • Creates new cache task implementation: Adds comprehensive task_cache.go with full CRUD operations (create, list, get, update, delete) for cached content
  • Enhances multimedia processing: Refactors common file processing logic into reusable functions with optimized memory allocation
  • Adds comprehensive test coverage: Includes extensive test suites for File API functionality, error handling, and cache operations

Key Features Implemented

🔄 File API Integration

  • Smart routing: Total request size calculation determines File API vs inline usage
  • Video file handling: All cache video files use File API regardless of size
  • File state management: Polls uploaded files until ACTIVE status before processing
  • Automatic cleanup: Uploaded files are deleted immediately after cache creation

📋 Cache Operations

  • Create: Build cached content with multimedia support and File API integration
  • List: Paginated listing of cached contents with configurable page size
  • Get: Retrieve specific cached content by name
  • Update: Modify cache expiration (TTL or expire time)
  • Delete: Remove cached content

⚡ Performance Optimizations

  • Memory pre-allocation: Optimized slice capacity for better performance
  • Unified processing: Consolidated media processing functions reduce code duplication
  • Efficient file handling: Streamlined upload and processing workflow

Technical Implementation

File API Workflow

// 1. Calculate total request size
totalSize := calculateTotalRequestSize(input)
useFileAPI := totalSize > MaxInlineSize || (isCache && hasVideos)

// 2. Upload files and wait for ACTIVE state
uploadedFiles, err := uploadFileAndWait(ctx, client, fileData, timeout)

// 3. Use FileData parts in content generation
parts = append(parts, genai.Part{FileData: &genai.FileData{
    FileURI: uploadedFile.URI,
    MIMEType: uploadedFile.MIMEType,
}})

// 4. Cleanup uploaded files after processing
defer cleanup(uploadedFiles)

Cache Configuration Support

  • Expiration control: TTL and expire time configuration
  • System instructions: Support for system messages and instructions
  • Tool integration: Function calling and tool configuration
  • Content management: Direct content specification or multimedia input processing

Files Added/Modified

New Files

  • pkg/component/ai/gemini/v0/task_cache.go - Complete cache operations implementation
  • pkg/component/ai/gemini/v0/task_cache_test.go - Comprehensive test coverage

Enhanced Files

  • pkg/component/ai/gemini/v0/common.go - File API integration and shared utilities
  • pkg/component/ai/gemini/v0/task_chat.go - Updated to use File API for large files
  • pkg/component/ai/gemini/v0/io.go - Enhanced input/output structures
  • Configuration files updated for new cache task support

Backward Compatibility

✅ Fully backward compatible - existing chat functionality unchanged
✅ Automatic fallback - small files continue using inline encoding
✅ Transparent optimization - File API usage is automatic based on size
✅ No API changes - existing interfaces remain identical

Benefits

  • Removes 20MB file size limitation for Gemini processing
  • Enables video caching capabilities with proper File API handling
  • Improves reliability with robust file state management
  • Enhances performance with optimized memory allocation and processing
  • Provides comprehensive cache management for advanced use cases

This implementation enables full-scale multimedia processing with Gemini AI while maintaining optimal performance and reliability through intelligent file routing and lifecycle management.

@pinglin pinglin merged commit b51c8f4 into main Sep 22, 2025
6 checks passed
@pinglin pinglin deleted the pinglin/feat-gemini-file-api-integration branch September 22, 2025 13:15
This was referenced Sep 22, 2025
Merged
Merged
jvallesm added a commit to instill-ai/instill-core that referenced this pull request Sep 23, 2025
@droplet-bot @jvallesm
chore(env): update PIPELINE_BACKEND_VERSION (#1405)
04cf805 
Because
- The version of the pipeline-backend service is not updated in the
instill-core repository.

This commit
- updates the `PIPELINE_BACKEND_VERSION` in the `.env` file to
`1b4cd1f`.
- updates the `pipelineBackend.image.tag` in the helm chart values.yaml
file to `1b4cd1f`.

## Changes in pipeline-backend
- fix(text): correct positions on duplicate markdown chunks
(instill-ai/pipeline-backend#1120)
- refactor(component,generic,http): replace env-based URL validation
with constructor injection (instill-ai/pipeline-backend#1121)
- fix(usage): add missing error filtering for users/admin
(instill-ai/pipeline-backend#1119)
- feat(component,ai,gemini): implement File API support for large files…
(instill-ai/pipeline-backend#1118)
- perf(data): optimize struct marshaling/unmarshaling with caching and …
(instill-ai/pipeline-backend#1117)
- feat(data): enhance unmarshaler with JSON string to struct conversion
(instill-ai/pipeline-backend#1116)
- feat(data): implement time types support with pattern validation
(instill-ai/pipeline-backend#1115)
- feat(component,ai,gemini): add multimedia support with unified format…
(instill-ai/pipeline-backend#1114)
- ci(workflows): adopt GitHub-hosted runner
(instill-ai/pipeline-backend#1113)
- perf(data): enhance comprehensive format coverage and optimize test
performance (instill-ai/pipeline-backend#1112)
- ci(workflows): adopt loarger runner for coverage test
(instill-ai/pipeline-backend#1111)
- perf(component,operator,document): optimize unit tests and fix
LibreOffice dependency failures (instill-ai/pipeline-backend#1110)
- perf(component,operator,video): optimize unit test performance by
59.7% (instill-ai/pipeline-backend#1109)
- perf(component,operator,image): optimize unit tests for 98.5% faster …
(instill-ai/pipeline-backend#1107)
- ci(docker): optimize Dockerfiles with multi-stage builds for faster
build times (instill-ai/pipeline-backend#1108)
- perf(data): implement automatic field naming convention detection with
LRU caching (instill-ai/pipeline-backend#1105)
- feat(component,ai,gemini): enhance streaming to output all fields
(instill-ai/pipeline-backend#1106)
- fix(component,ai,gemini): correct text-based documents logic
(instill-ai/pipeline-backend#1103)
- test(component,generic,http): replace external httpbin.org dependency
with local test server (instill-ai/pipeline-backend#1101)
- ci(docker): add GitHub fallback for ffmpeg installation
(instill-ai/pipeline-backend#1102)

Co-authored-by: jvallesm <3977183+jvallesm@users.noreply.github.com>
jvallesm pushed a commit that referenced this pull request Oct 7, 2025
@droplet-bot
chore(main): release 0.61.0 (#1104)
1073de3 
🤖 I have created a release *beep* *boop*
---


##
[0.61.0](v0.60.0...v0.61.0)
(2025-10-06)


### Features

* **component,ai,gemini:** add image generation support
([#1122](#1122))
([d986614](d986614))
* **component,ai,gemini:** add multimedia support with unified format…
([#1114](#1114))
([291b379](291b379))
* **component,ai,gemini:** add text embeddings task support
([#1129](#1129))
([d7ca6cf](d7ca6cf))
* **component,ai,gemini:** enhance streaming to output all fields
([#1106](#1106))
([dfb6b24](dfb6b24))
* **component,ai,gemini:** implement automatic format conversion for
unsupported media types
([#1128](#1128))
([f767b8a](f767b8a))
* **component,ai,gemini:** implement File API support for large files…
([#1118](#1118))
([b51c8f4](b51c8f4))
* **data:** add comprehensive AVIF image format support
([#1135](#1135))
([76d6941](76d6941))
* **data:** add HEIC/HEIF image support and normalize MIME types
([#1127](#1127))
([2dfa254](2dfa254))
* **data:** enhance unmarshaler with JSON string to struct conversion
([#1116](#1116))
([9e06b7c](9e06b7c))
* **data:** implement time types support with pattern validation
([#1115](#1115))
([79630c0](79630c0))


### Bug Fixes

* **compogen:** escape curly braces for readme.com compatibility
([#1124](#1124))
([904992d](904992d))
* **component,ai,gemini:** add operation validation for cache task
([#1130](#1130))
([9e19255](9e19255))
* **component,ai,gemini:** correct text-based documents logic
([#1103](#1103))
([ed5a111](ed5a111))
* **component,ai,gemini:** unify InlineData processing and enable images
in streaming responses
([#1125](#1125))
([3117046](3117046))
* **data:** remove duplicate dot in generated filenames
([#1136](#1136))
([0a74a00](0a74a00))
* **external:** fix Content-Disposition header parsing for filename
extraction
([#1132](#1132))
([869b081](869b081))
* **service:** handle null JSON metadata in pipeline conversion
([#1134](#1134))
([b244784](b244784))
* **text:** correct positions on duplicate markdown chunks
([#1120](#1120))
([1b4cd1f](1b4cd1f))
* **usage:** add missing error filtering for users/admin
([#1119](#1119))
([cd1bd55](cd1bd55))


### Refactor

* **component,ai,gemini:** merge usage and usage-metadata fields into
single usage field
([#1126](#1126))
([a6046cd](a6046cd))
* **component,ai.gemini:** standardize file api timeout and use native
embedding type
([#1133](#1133))
([174f7d6](174f7d6))
* **component,generic,http:** move test functions to test files and
improve code legibility
([#1131](#1131))
([1153a09](1153a09))
* **component,generic,http:** replace env-based URL validation with
constructor injection
([#1121](#1121))
([f1f7d2f](f1f7d2f))


### Tests

* **component,generic,http:** replace external httpbin.org dependency
with local test server
([#1101](#1101))
([a82d155](a82d155))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@droplet-bot droplet-bot mentioned this pull request Oct 7, 2025
Merged
donch1989 pushed a commit that referenced this pull request Oct 7, 2025
@droplet-bot
chore(main): release 0.61.0 (#1137)
3344938 
🤖 I have created a release *beep* *boop*
---


##
[0.61.0](v0.60.0...v0.61.0)
(2025-10-07)


### Features

* **component,ai,gemini:** add image generation support
([#1122](#1122))
([d986614](d986614))
* **component,ai,gemini:** add multimedia support with unified format…
([#1114](#1114))
([291b379](291b379))
* **component,ai,gemini:** add text embeddings task support
([#1129](#1129))
([d7ca6cf](d7ca6cf))
* **component,ai,gemini:** enhance streaming to output all fields
([#1106](#1106))
([dfb6b24](dfb6b24))
* **component,ai,gemini:** implement automatic format conversion for
unsupported media types
([#1128](#1128))
([f767b8a](f767b8a))
* **component,ai,gemini:** implement File API support for large files…
([#1118](#1118))
([b51c8f4](b51c8f4))
* **data:** add comprehensive AVIF image format support
([#1135](#1135))
([76d6941](76d6941))
* **data:** add HEIC/HEIF image support and normalize MIME types
([#1127](#1127))
([2dfa254](2dfa254))
* **data:** enhance unmarshaler with JSON string to struct conversion
([#1116](#1116))
([9e06b7c](9e06b7c))
* **data:** implement time types support with pattern validation
([#1115](#1115))
([79630c0](79630c0))


### Bug Fixes

* **compogen:** escape curly braces for readme.com compatibility
([#1124](#1124))
([904992d](904992d))
* **component,ai,gemini:** add operation validation for cache task
([#1130](#1130))
([9e19255](9e19255))
* **component,ai,gemini:** correct text-based documents logic
([#1103](#1103))
([ed5a111](ed5a111))
* **component,ai,gemini:** unify InlineData processing and enable images
in streaming responses
([#1125](#1125))
([3117046](3117046))
* **component,document:** fix incorrect expected value in the unit test
([#1138](#1138))
([189dbd6](189dbd6))
* **data:** remove duplicate dot in generated filenames
([#1136](#1136))
([0a74a00](0a74a00))
* **external:** fix Content-Disposition header parsing for filename
extraction
([#1132](#1132))
([869b081](869b081))
* **service:** handle null JSON metadata in pipeline conversion
([#1134](#1134))
([b244784](b244784))
* **text:** correct positions on duplicate markdown chunks
([#1120](#1120))
([1b4cd1f](1b4cd1f))
* **usage:** add missing error filtering for users/admin
([#1119](#1119))
([cd1bd55](cd1bd55))


### Miscellaneous

* release v0.61.0
([e1db93c](e1db93c))


### Refactor

* **component,ai,gemini:** merge usage and usage-metadata fields into
single usage field
([#1126](#1126))
([a6046cd](a6046cd))
* **component,ai.gemini:** standardize file api timeout and use native
embedding type
([#1133](#1133))
([174f7d6](174f7d6))
* **component,generic,http:** move test functions to test files and
improve code legibility
([#1131](#1131))
([1153a09](1153a09))
* **component,generic,http:** replace env-based URL validation with
constructor injection
([#1121](#1121))
([f1f7d2f](f1f7d2f))


### Tests

* **component,generic,http:** replace external httpbin.org dependency
with local test server
([#1101](#1101))
([a82d155](a82d155))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@donch1989 donch1989 Awaiting requested review from donch1989 donch1989 is a code owner

@jvallesm jvallesm Awaiting requested review from jvallesm jvallesm is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pinglin