Skip to content

Search analyzer fingerprint pins index to exact ICU patch version; mismatch error suggests a dangerous rebuild #75

Description

@58bits

Summary

The search analyzer fingerprint (packages/search-analysis — analyzer.js builds it from process.versions.icu) pins the index to the exact ICU patch version of the Node process that wrote it. Any client on a different Node patch release is then locked out of writing to the index — even when the ICU difference almost certainly has no effect on tokenization.

What happened

Running import-docs.ts from a workstation (Node v24.15.0, ICU 78.2) against a production database whose index was written by the app server (Node ≥ 24.16.0, ICU 78.3) fails on every document:

Search index for collection "docs" uses "portable1+nfkc-lower1+icu78.3+locale1+default-en+han-zh+identifiers2+han-bigram1";
expected analyzer "portable1+nfkc-lower1+icu78.2+locale1+default-en+han-zh+identifiers2+han-bigram1".
Rebuild this collection's search index.

Node bumped ICU 78.2 → 78.3 in v24.16.0 (nodejs/node#62324), so this now bites anyone whose local Node patch version differs from the server's — a routine situation.

Why the current behavior is risky, not just inconvenient

The error's advice — "Rebuild this collection's search index" — is actively dangerous when the mismatched party is a client machine. Rebuilding from the workstation would restamp the production index as icu78.2, and the production app would then fail this same check on every write. The safe fix (align the client's Node version with the server's) is not mentioned.

Ideas to consider

  1. Improve the error message: state that the ICU component comes from the running Node version, and that the usual fix is to run the writer with the same Node/ICU as whatever owns the index — rebuild only from the serving environment.
  2. Reconsider fingerprint granularity: ICU patch releases (78.2 → 78.3) rarely change segmentation behavior. Options: fingerprint only the ICU major version, or keep strict matching but downgrade patch-level mismatch to a warning (possibly behind a flag).
  3. Escape hatch: an explicit opt-in (env var or CLI flag) to accept a fingerprint mismatch for one-off imports, clearly documented as "you accept possibly inconsistent tokenization until the next rebuild".
  4. Docs: note in the import/search docs that offline writers (import scripts, CI) must match the serving app's Node version.

Environment

  • @byline/search-postgres 4.12.0 / @byline/search-analysis 4.12.0
  • Writer: macOS, Node v24.15.0 (ICU 78.2); index owner: production app on Node ≥ 24.16.0 (ICU 78.3)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions