Skip to content

feat(data): enhance unmarshaler with JSON string to struct conversion #1116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 22, 2025
Merged

feat(data): enhance unmarshaler with JSON string to struct conversion #1116

merged 1 commit into from
Sep 22, 2025

Conversation

@pinglin
Copy link
Member

@pinglin pinglin commented Sep 22, 2025

Because

  • YAML multiline strings containing JSON schemas (like response-schema: |) could not be automatically converted to Go struct types like genai.Schema

This commit

  • Adds automatic JSON string → struct conversion capability to the core unmarshaling framework
  • Implements fast pre-check (stringValue[0] == '{' or '[') to avoid unnecessary JSON parsing overhead on regular strings
  • Adds tryUnmarshalJSONString() helper function that uses direct json.Unmarshal to preserve complex nested structures

@pinglin pinglin merged commit 9e06b7c into main Sep 22, 2025
5 of 6 checks passed
@pinglin pinglin deleted the pinglin/feat-data-json-string-struct-conversion branch September 22, 2025 03:17
This was referenced Sep 22, 2025
Merged
Merged
jvallesm added a commit to instill-ai/instill-core that referenced this pull request Sep 23, 2025
@droplet-bot @jvallesm
chore(env): update PIPELINE_BACKEND_VERSION (#1405)
04cf805 
Because
- The version of the pipeline-backend service is not updated in the
instill-core repository.

This commit
- updates the `PIPELINE_BACKEND_VERSION` in the `.env` file to
`1b4cd1f`.
- updates the `pipelineBackend.image.tag` in the helm chart values.yaml
file to `1b4cd1f`.

## Changes in pipeline-backend
- fix(text): correct positions on duplicate markdown chunks
(instill-ai/pipeline-backend#1120)
- refactor(component,generic,http): replace env-based URL validation
with constructor injection (instill-ai/pipeline-backend#1121)
- fix(usage): add missing error filtering for users/admin
(instill-ai/pipeline-backend#1119)
- feat(component,ai,gemini): implement File API support for large files…
(instill-ai/pipeline-backend#1118)
- perf(data): optimize struct marshaling/unmarshaling with caching and …
(instill-ai/pipeline-backend#1117)
- feat(data): enhance unmarshaler with JSON string to struct conversion
(instill-ai/pipeline-backend#1116)
- feat(data): implement time types support with pattern validation
(instill-ai/pipeline-backend#1115)
- feat(component,ai,gemini): add multimedia support with unified format…
(instill-ai/pipeline-backend#1114)
- ci(workflows): adopt GitHub-hosted runner
(instill-ai/pipeline-backend#1113)
- perf(data): enhance comprehensive format coverage and optimize test
performance (instill-ai/pipeline-backend#1112)
- ci(workflows): adopt loarger runner for coverage test
(instill-ai/pipeline-backend#1111)
- perf(component,operator,document): optimize unit tests and fix
LibreOffice dependency failures (instill-ai/pipeline-backend#1110)
- perf(component,operator,video): optimize unit test performance by
59.7% (instill-ai/pipeline-backend#1109)
- perf(component,operator,image): optimize unit tests for 98.5% faster …
(instill-ai/pipeline-backend#1107)
- ci(docker): optimize Dockerfiles with multi-stage builds for faster
build times (instill-ai/pipeline-backend#1108)
- perf(data): implement automatic field naming convention detection with
LRU caching (instill-ai/pipeline-backend#1105)
- feat(component,ai,gemini): enhance streaming to output all fields
(instill-ai/pipeline-backend#1106)
- fix(component,ai,gemini): correct text-based documents logic
(instill-ai/pipeline-backend#1103)
- test(component,generic,http): replace external httpbin.org dependency
with local test server (instill-ai/pipeline-backend#1101)
- ci(docker): add GitHub fallback for ffmpeg installation
(instill-ai/pipeline-backend#1102)

Co-authored-by: jvallesm <3977183+jvallesm@users.noreply.github.com>
jvallesm pushed a commit that referenced this pull request Oct 7, 2025
@droplet-bot
chore(main): release 0.61.0 (#1104)
1073de3 
🤖 I have created a release *beep* *boop*
---


##
[0.61.0](v0.60.0...v0.61.0)
(2025-10-06)


### Features

* **component,ai,gemini:** add image generation support
([#1122](#1122))
([d986614](d986614))
* **component,ai,gemini:** add multimedia support with unified format…
([#1114](#1114))
([291b379](291b379))
* **component,ai,gemini:** add text embeddings task support
([#1129](#1129))
([d7ca6cf](d7ca6cf))
* **component,ai,gemini:** enhance streaming to output all fields
([#1106](#1106))
([dfb6b24](dfb6b24))
* **component,ai,gemini:** implement automatic format conversion for
unsupported media types
([#1128](#1128))
([f767b8a](f767b8a))
* **component,ai,gemini:** implement File API support for large files…
([#1118](#1118))
([b51c8f4](b51c8f4))
* **data:** add comprehensive AVIF image format support
([#1135](#1135))
([76d6941](76d6941))
* **data:** add HEIC/HEIF image support and normalize MIME types
([#1127](#1127))
([2dfa254](2dfa254))
* **data:** enhance unmarshaler with JSON string to struct conversion
([#1116](#1116))
([9e06b7c](9e06b7c))
* **data:** implement time types support with pattern validation
([#1115](#1115))
([79630c0](79630c0))


### Bug Fixes

* **compogen:** escape curly braces for readme.com compatibility
([#1124](#1124))
([904992d](904992d))
* **component,ai,gemini:** add operation validation for cache task
([#1130](#1130))
([9e19255](9e19255))
* **component,ai,gemini:** correct text-based documents logic
([#1103](#1103))
([ed5a111](ed5a111))
* **component,ai,gemini:** unify InlineData processing and enable images
in streaming responses
([#1125](#1125))
([3117046](3117046))
* **data:** remove duplicate dot in generated filenames
([#1136](#1136))
([0a74a00](0a74a00))
* **external:** fix Content-Disposition header parsing for filename
extraction
([#1132](#1132))
([869b081](869b081))
* **service:** handle null JSON metadata in pipeline conversion
([#1134](#1134))
([b244784](b244784))
* **text:** correct positions on duplicate markdown chunks
([#1120](#1120))
([1b4cd1f](1b4cd1f))
* **usage:** add missing error filtering for users/admin
([#1119](#1119))
([cd1bd55](cd1bd55))


### Refactor

* **component,ai,gemini:** merge usage and usage-metadata fields into
single usage field
([#1126](#1126))
([a6046cd](a6046cd))
* **component,ai.gemini:** standardize file api timeout and use native
embedding type
([#1133](#1133))
([174f7d6](174f7d6))
* **component,generic,http:** move test functions to test files and
improve code legibility
([#1131](#1131))
([1153a09](1153a09))
* **component,generic,http:** replace env-based URL validation with
constructor injection
([#1121](#1121))
([f1f7d2f](f1f7d2f))


### Tests

* **component,generic,http:** replace external httpbin.org dependency
with local test server
([#1101](#1101))
([a82d155](a82d155))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
@droplet-bot droplet-bot mentioned this pull request Oct 7, 2025
Merged
donch1989 pushed a commit that referenced this pull request Oct 7, 2025
@droplet-bot
chore(main): release 0.61.0 (#1137)
3344938 
🤖 I have created a release *beep* *boop*
---


##
[0.61.0](v0.60.0...v0.61.0)
(2025-10-07)


### Features

* **component,ai,gemini:** add image generation support
([#1122](#1122))
([d986614](d986614))
* **component,ai,gemini:** add multimedia support with unified format…
([#1114](#1114))
([291b379](291b379))
* **component,ai,gemini:** add text embeddings task support
([#1129](#1129))
([d7ca6cf](d7ca6cf))
* **component,ai,gemini:** enhance streaming to output all fields
([#1106](#1106))
([dfb6b24](dfb6b24))
* **component,ai,gemini:** implement automatic format conversion for
unsupported media types
([#1128](#1128))
([f767b8a](f767b8a))
* **component,ai,gemini:** implement File API support for large files…
([#1118](#1118))
([b51c8f4](b51c8f4))
* **data:** add comprehensive AVIF image format support
([#1135](#1135))
([76d6941](76d6941))
* **data:** add HEIC/HEIF image support and normalize MIME types
([#1127](#1127))
([2dfa254](2dfa254))
* **data:** enhance unmarshaler with JSON string to struct conversion
([#1116](#1116))
([9e06b7c](9e06b7c))
* **data:** implement time types support with pattern validation
([#1115](#1115))
([79630c0](79630c0))


### Bug Fixes

* **compogen:** escape curly braces for readme.com compatibility
([#1124](#1124))
([904992d](904992d))
* **component,ai,gemini:** add operation validation for cache task
([#1130](#1130))
([9e19255](9e19255))
* **component,ai,gemini:** correct text-based documents logic
([#1103](#1103))
([ed5a111](ed5a111))
* **component,ai,gemini:** unify InlineData processing and enable images
in streaming responses
([#1125](#1125))
([3117046](3117046))
* **component,document:** fix incorrect expected value in the unit test
([#1138](#1138))
([189dbd6](189dbd6))
* **data:** remove duplicate dot in generated filenames
([#1136](#1136))
([0a74a00](0a74a00))
* **external:** fix Content-Disposition header parsing for filename
extraction
([#1132](#1132))
([869b081](869b081))
* **service:** handle null JSON metadata in pipeline conversion
([#1134](#1134))
([b244784](b244784))
* **text:** correct positions on duplicate markdown chunks
([#1120](#1120))
([1b4cd1f](1b4cd1f))
* **usage:** add missing error filtering for users/admin
([#1119](#1119))
([cd1bd55](cd1bd55))


### Miscellaneous

* release v0.61.0
([e1db93c](e1db93c))


### Refactor

* **component,ai,gemini:** merge usage and usage-metadata fields into
single usage field
([#1126](#1126))
([a6046cd](a6046cd))
* **component,ai.gemini:** standardize file api timeout and use native
embedding type
([#1133](#1133))
([174f7d6](174f7d6))
* **component,generic,http:** move test functions to test files and
improve code legibility
([#1131](#1131))
([1153a09](1153a09))
* **component,generic,http:** replace env-based URL validation with
constructor injection
([#1121](#1121))
([f1f7d2f](f1f7d2f))


### Tests

* **component,generic,http:** replace external httpbin.org dependency
with local test server
([#1101](#1101))
([a82d155](a82d155))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@donch1989 donch1989 Awaiting requested review from donch1989 donch1989 is a code owner

@jvallesm jvallesm Awaiting requested review from jvallesm jvallesm is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pinglin