Skip to content

Extract text and annotations from Shamela book pages - #403

Open
XIAX66 wants to merge 1 commit into
kepano:mainfrom
XIAX66:fix/shamela-book-root
Open

XIAX66 wants to merge 1 commit into
kepano:mainfrom
XIAX66:fix/shamela-book-root

Conversation

@XIAX66

@XIAX66 XIAX66 commented Oct 2, 2026

Copy link
Copy Markdown

The book page reported in #386 extracts the search panel's 73 words while dropping all five book and annotation paragraphs. The adjacent page has the same failure. Both pages keep their book text in a unique .nass container; the search panel reuses the generic #content entry point.

Add a Shamela extractor that selects the known text container through the existing contentSelector pipeline. It applies only to shamela.ws or www.shamela.ws, numeric /book/<book>/<page> paths, and a unique nonempty .nass root. Missing, empty, or ambiguous roots retain automatic extraction. An explicit caller contentSelector opts out of the new extractor. Generic hidden-content recovery is unchanged, including articles rendered inside hidden overlays/dialogs.

Related to #386. This targets the verified book-page structure; it does not establish support for every Shamela layout or change cross-site dialog heuristics.

Validation on Node 24 / macOS:

  • Two anonymized HTML/expected-output fixtures, with 20-word and 202-word chapters, fail before the fix and pass afterward. Book/page identifiers are placeholders; the functional host is retained for routing.
  • Regression cases cover both sides of the 50-word retry threshold, nested hidden ancestors, active/closed overlays, duplicate IDs, caller selection, URL/root boundaries, and existing hidden/expandable article recovery. Poisoned-attribute cases check sanitization with optional cleanup passes disabled.
  • On the downloaded reported page and its adjacent page, HTML and rendered Markdown retain 5/5 and 10/10 book/annotation paragraphs respectively under linkedom and JSDOM, with debug enabled or disabled. Normal output contains 215 and 175 words. Offline extraction uses the cached HTML.
  • npm test and npm run test:jsdom: 800 tests pass in each. npm run build, npm run lint:innerhtml, npm run lint:unused, npm run size, and git diff --check pass.
  • Production core/full bundles pass 34 checks using native DOM in an isolated headless browser. The compiled CLI retains all five paragraphs on the reported page.

Cloudflare Worker, happy-dom, and Node 20/22 were not directly exercised locally. Remote CI has not yet been verified.

Prepared with AI assistance; the reproduction and validation above were run locally.

@XIAX66
XIAX66 marked this pull request as ready for review October 3, 2026 14:24
szepeviktor pushed a commit to szepeviktor/defuddle that referenced this pull request Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant