Skip to content

feat: add title field and metadata handling for ZIM file creation #855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 22, 2025

Conversation

nicolapace
Copy link
Contributor

Fixes #677

I added a Title field in the Zim creation page. This field is automatically filled with the Selection Title reduced to 30 characters. In this way users can change it if they want. This field is required and is of max 30 chars. (I also added a char counter for both the title and the description).

I also added backend validation for the size of zim metadata (title (30) and description (80) as per https://wiki.openzim.org/wiki/Metadata).

I added new unit test for the new "title field" and the validation, for both the backend and the frontend. I also fixed some unit tests and functions to support the new zimtitle field.

Here is the Zim creation page with the new field:

Screenshot

@kelson42
Copy link
Collaborator

@nicolapace Thank you for your PR! But the unity for counting is not char, but lexem. There is a javascript/typescript implementation of such a counter in openzim/mwoffliner.

@kelson42 kelson42 requested review from audiodude and kelson42 April 15, 2025 13:18
@audiodude
Copy link
Member

@nicolapace Thank you for your PR! But the unity for counting is not char, but lexem. There is a javascript/typescript implementation of such a counter in openzim/mwoffliner.

#677 (comment)

What is a lexem and how does it compare to a grapheme? I feel like I'm going to have to read a Chomsky book to implement this PR. 🙃

Copy link
Member

@audiodude audiodude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good, great work!

I think the handling of the error conditions on the backend needs to be improved however.

@kelson42
Copy link
Collaborator

kelson42 commented Apr 16, 2025

@audiodude I meant grapheme, sorry. See link given in the issue comment.

@nicolapace nicolapace force-pushed the limit-zimtitle-length branch from 90a214e to 32b3b55 Compare April 17, 2025 13:22
@nicolapace
Copy link
Contributor Author

@audiodude @kelson42 @benoit74 Thanks for the feedback and the suggestions!

I pushed a new commit that addresses the discussed issues, in particular:

  • I am now using split-by-grapheme npm package and regex python module to check for num. of graphemes in both frontend and backend as suggested by @kelson42 and @benoit74
  • as suggested by @audiodude I changed the validate_zim_metadata() function to Raise exceptions directly and I created new exceptions
  • I also added validation for the long_description field as described in https://wiki.openzim.org/wiki/Metadata#Graphemes
  • I disable the submit button if inputs are not valid
  • I changed unit tests to support the new changes and added some new unit tests (for the long description validation and the handling of graphemes)

(I pushed a new commit instead of amending the last one for better clarity. We can squash them later if you prefer.)

@nicolapace nicolapace force-pushed the limit-zimtitle-length branch from 32b3b55 to 80d06a4 Compare April 17, 2025 13:35
@nicolapace nicolapace requested a review from audiodude April 17, 2025 13:44
Copy link
Member

@audiodude audiodude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, just a few small comments.

- Add split-by-grapheme npm package to handle grapheme counting accurately.
- Modify ZimFile.vue to implement grapheme counting for title, description, and long description fields
- Introduce new exceptions for invalid ZIM metadata: InvalidZimTitleError, InvalidZimDescriptionError, and InvalidZimLongDescriptionError.
- Add regex python module to Pipfile and rebuilt Pipfile.lock
- Enhance backend logic in builder.py and zimfarm.py to validate ZIM metadata against grapheme limits using regex module
- Update unit tests to cover new validation rules and exceptions for ZIM metadata.
- Update ZIM file creation tests to check for disabled state of the request button
- Add frontend and backend validation of long description (max. length, min. length and difference to description)
@nicolapace nicolapace force-pushed the limit-zimtitle-length branch from 907a221 to 4d30fbd Compare April 22, 2025 18:53
@nicolapace nicolapace requested a review from audiodude April 22, 2025 18:54
Copy link
Member

@audiodude audiodude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

Copy link

codecov bot commented Apr 22, 2025

Codecov Report

Attention: Patch coverage is 97.14286% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.63%. Comparing base (9a19180) to head (4d30fbd).
Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
wp1/zimfarm.py 95.45% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #855      +/-   ##
==========================================
+ Coverage   92.58%   92.63%   +0.05%     
==========================================
  Files          66       66              
  Lines        3546     3573      +27     
==========================================
+ Hits         3283     3310      +27     
  Misses        263      263              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@audiodude audiodude merged commit 15ac523 into openzim:main Apr 22, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Limit length of ZIM file zimTitle field
3 participants