fix(embeddings): accept RFC 2045 last-line image folds - #680
Closed
cursor[bot] wants to merge 4 commits into
Closed
Conversation
Token-budget map/reduce still averages provider parts of one input. Buyers searching naruon invoice mail need the balance line as its own vector. chunking_strategy=meaning_units expands /v1/batch/embeddings into email, HTML, image, and paragraph units and returns chunk_units with source offsets. Omit keeps the naruon one-vector-per-input contract. Grounded in Zhao et al. (2024), Qu et al. (2025), UAX #29, and Lewis et al. (2020). Next action: POST the raw invoice email with meaning_units and search chunk_units for the invoice id. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Keep innermost HTML leaves so a Gmail wrapper div does not glue the greeting to the invoice line. Accept charset parameters, URL-safe payloads, and MIME line wraps on data:image spans so a scanned invoice keeps one source offset. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
The {16,} continuation floor rejected a 76-column wrap whose last
line is 15 alphabet characters plus padding, so leftover base64
glued onto the invoice unit. Walk the payload instead: stop at
padding, space, quote, or >, and keep a following alphanumeric
invoice line out of the image.
Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
seonghobae
marked this pull request as ready for review
August 16, 2026 17:05
seonghobae
enabled auto-merge (squash)
August 16, 2026 17:05
Author
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
auto-merge was automatically disabled
August 16, 2026 18:23
Pull request was closed
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Buyer next action
Do not merge #665. Merge this successor instead. POST
/v1/batch/embeddingswith a 76-column RFC 2045 wrap of adata:imagescan (the in-repo 1x1 PNG split after 76 base64 characters) and"chunking_strategy": "meaning_units". Searchchunk_unitsforINV-20260816— leftoverAAAASUVORK5CYII=must not be in that vector. Wrapped<div><p>greeting</p><p>invoice</p></div>isolation from #665 is kept.Why
#665 (
ef4370e) used a{16,}continuation floor counted before padding. RFC 2045 wrap of the 92-character PNG leavesAAAASUVORK5CYII=(15 alphabet characters +=). The continuation was rejected; leftover base64 glued onto the invoicebody_paragraph. The in-repo MIME fixture wrapped mid-payload so line 2 stayed long — false green.This tip walks the payload: stop at padding / space / quote /
>, fold only a following line that is entirely base64/padding, and keep a following alphanumeric invoice line out of the image.Tests
RFC 2045 column-76, short padded last line, same-line prose after
=, and HTTP POST of the wrap all isolate the invoice unit.References
Freed, N., & Borenstein, N. (1996). Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies (RFC 2045). RFC Editor. https://doi.org/10.17487/RFC2045
Masinter, L. (1998). The "data" URL scheme (RFC 2397). RFC Editor. https://doi.org/10.17487/RFC2397
Independent non-author APPROVE is required. This automation will not self-approve or merge. Do not fold live NIM OCR, Responses SSE, or KV allowlist here.