Conversation
- Switch metadata listing to .sorted(String.CASE_INSENSITIVE_ORDER) in TikaDocParser - Reorder the raw-metadata assertions in TikaDocParserTest to match the new order - Drop a redundant duplicate xmpDM:genre entry in the mp3 assertions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… HTML metadata attribute names
…Tesseract - Tika 4 removed the static ExternalParser.check(); replace the Tesseract availability probe with the same hasTesseract() check used in production - Convert the runtime Assumptions skip into a declarative @DisabledIf so the test reports as disabled (with a reason) instead of aborted - Drop the now-unused Assumptions import Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ika 4 - Refresh PDF (47 keys) and MP3 (20 keys) raw metadata examples in local-fs to match Tika 4 output (X-TIKA:*, Content-Type-Magic-Detected, pdf:docinfo:*, access_permission:*), sorted case-insensitively like the real output - Fix the REST examples in rest.rst (X-TIKA:Parsed-By and friends) - Warn that fs.tika_config_path is temporarily unsupported on Tika 4 (XML configuration mechanism removed upstream); note it in the 2.10 changelog Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mespace - Tika 4 namespaces all metadata-extractor image keys under "img:" (both the bare keys like "Number of Tables" and the "Exif IFD0:/SubIFD:" ones); values are unchanged, ICC:/tiff:/Content-*/X-TIKA: keys stay as-is - Bump hasSize(64) -> hasSize(66) (Tika 4 adds Content-Type-Magic-Detected and Content-Length) and add the img: prefixes, verified against real output - Sort the raw-metadata assertions case-insensitively for readability Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Tika 4 renamed the resourceName metadata key to X-TIKA:resourceName; update the JsonPath in documentWithExternalTags to bracket notation for the new key Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Note the img: namespace for image/EXIF metadata keys - Note resourceName -> X-TIKA:resourceName rename - Note new keys: Content-Type-Magic-Detected, X-TIKA:Parsed-By-Full-Set and the text encoding-detection keys (X-TIKA:detectedEncoding/encodingDetectionTrace/encodingDetector) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates FSCrawler from Apache Tika 3.3.1 to Tika 4.0.0-beta-1.
tika.versionto4.0.0-beta-1and marktika-parsers-standard-packageas<type>pom</type>(it is a POM aggregator in Tika 4).TikaInstancefor the Tika 4 API:Parser.parse()now requires aTikaInputStream(wrapped inextractText).TesseractOCRConfig; the parser is built from that config.PDFParser.setOcrStrategy(String)toPDFParserConfig.setOcr(OcrConfig)with an enum (FSCrawler string values are mapped,ocr_and_textstays the default).TikaCoreProperties.RESOURCE_NAME_KEYinstead of the literalresourceName(renamed toX-TIKA:resourceName).ExternalParser.check()was removed in Tika 4; the OCR IT now skips via@DisabledIf+TesseractOCRParser.hasTesseract().Breaking changes
fs.tika_config_pathis temporarily unsupported: Tika 4 removed the XML configuration-file mechanism. FSCrawler now fails fast with a clear message if the setting is used. Support is expected to return on top of Tika 4's JSON configuration.meta.raw.*metadata key names changed (documented in the 2.10 release notes):img:(e.g.Number of Tables→img:Number of Tables,Exif IFD0:*→img:Exif IFD0:*).resourceName→X-TIKA:resourceName.Content-Type-Magic-Detected,X-TIKA:Parsed-By-Full-Setand text encoding-detection keys.Documentation
tika_config_pathremoval and themeta.raw.*key changes in the 2.10 release notes.Test plan
mvn clean test -DskipIntegTests(tika module green).mvn clean install -DskipTests -Ddocker.skip(16 modules, enforcer + Spotless pass).FsCrawlerTestOcrIT(ocr + ocr_disabled) andFsCrawlerRestIT#documentWithExternalTagsgreen.img:/hasSizeregressions (RawIT and SubDirsIT unaffected).Known upstream issue
Follow-up
🤖 Generated with Claude Code
Note
High Risk
Core document parsing and
meta.raw.*field names change across the ingestion path; custom Tika XML config now fails startup, so deployments relying on those settings or fixed metadata keys need migration.Overview
Upgrades FSCrawler from Tika 3.3.1 to Tika 4.0.0-beta-1, including Maven dependency changes (
tika-parsers-standard-packageas a POM aggregator) and a rework ofTikaInstancefor the Tika 4 APIs.Parsing and OCR:
extractTextwraps inputs inTikaInputStream; Tesseract paths are applied viaTesseractOCRConfigand PDF OCR strategies map toOcrConfig.Strategy; resource names useTikaCoreProperties.RESOURCE_NAME_KEY.fs.tika_config_pathis rejected withFsCrawlerIllegalConfigurationExceptionbecause XML Tika config was removed upstream.Indexed metadata (
meta.raw.*): Keys change with Tika 4—e.g.resourceName→X-TIKA:resourceName, image/EXIF fields underimg:, plus new keys likeContent-Type-Magic-Detectedand encoding-detection fields. Docs, release notes, REST examples, and unit/integration tests are updated to match.Tests: OCR integration skips when Tesseract is missing via
TesseractOCRParser.hasTesseract()instead of removedExternalParser.check().Reviewed by Cursor Bugbot for commit a4cdd7c. Bugbot is set up for automated code reviews on this repo. Configure here.