Limit length of ZIM file `zimTitle` field

[audiodude]

Currently, the customZimTitle field that we send to the zimfarm is copied directly from the Builder name, which has no restrictions on its length, besides being truncated to 255 characters in the database. ZIM titles, on the other hand, can only be 30 characters max.

A couple of options:

1. Inherit the ZIM restriction and limit Builder titles to 30 characters
2. Truncate the Builder title when creating a ZIM

I tend to prefer the second option, because I don't see why the limitations of a ZIM should limit the entire WP1 UI.

[benoit74]

For me option 2 is not very good, because it means that we can get ZIM titles with a last word truncated in the middle (kind of ugly) and also duplicate ZIM titles (for different selections) due to identical first 30 chars among them.

I tend to prefer either option 1 (with a clear warning for the user) or a third option: add the ZIM title to information the user must input when creating a ZIM (just like description and long description) and automatically populate it based on the truncated Builder title. This way, the user will be able to do whatever is needed to make it look good.

[audiodude]

Yes I like this new "option 3" as well: query for a separate ZIM title on the ZIM creation page, that is auto populated with a (possibly truncated) version of the Builder name. Then the user can decide how best to untruncate it.

👍

[kelson42]

It is not 30 characters but 30 graphemes, see https://wiki.openzim.org/wiki/Metadata.

Description metadata should also be checked,

It is really important to implement this otherwise the system will fail to generate a ZIM file

[audiodude]

@kelson42

Sorry, I'm not sure what you mean by "grapheme". I've heard the word before, and read the Wikipedia article that is linked from the page you linked (https://wiki.openzim.org/wiki/Metadata), but that doesn't help me understand the distinction between a grapheme and a character.

I assume we're talking about the difference between bytes and characters? In ASCII, 1 character is always 1 byte, but in Unicode, a character can be multiple bytes. It would make sense that this is not a byte limit of 30, but a character limit of 30, so if the writing system is Chinese, it's still 30 characters which might be ~100 bytes.

On the other hand, a "grapheme limit" of 30 makes no sense to me. How are graphemes generally represented in Unicode?

I must say, I don't understand what a grapheme is or how it relates to bytes, characters and Unicode. Please explain further, or better yet, update the openzim wiki page with the information because I'm sure others in the future will be wondering the same thing.

[kelson42]

@audiodude I won't be able to explain better than Wikipedia what is a grapheme. IMHO what you miss is the tooling to count them as you don't understand how it relates to an encoding like UTF8.

To answer this latest question you can for example read https://stackoverflow.com/questions/27331819/whats-the-difference-between-a-character-a-code-point-a-glyph-and-a-grapheme#27331885. But to keep it short there is no straight way to know how many grpahemes you have based on the UTF8 string.

You probably better have to use a dedicated library like in MWoffliner (sorry don't have the exact reference, butbif you search for "grapheme" you will find it quickly).

This PR - like it is - will fail very often with title /descriptions in Non-latin alphabets. It will pass in WP1 and fail straight in MWoffliner.

[audiodude]

Thanks for the deeper explanation and the link. It sounds like you're referring to the old "number of bytes != number of characters" issue that I mentioned in my comment, except using the more precise term grapheme instead of character. I will reference the grapheme implementation specifically in MWoffliner, since that is the system we wish to integrate with.

[benoit74]

Node.JS library we use in mwoffliner is split-by-grapheme

You can find it in used at https://github.com/openzim/mwoffliner/blob/428b2084a2a52d3ea2cbd5839697e02983644925/src/util/metaData.ts#L27

[audiodude]

I think Python is okay right? Because Python str is inherently Unicode, so Python "characters" are actually "graphemes" and counting them gives the right number:

$ python
Python 3.12.0 (main, Oct  3 2023, 17:47:52) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s = '请给我一杯咖啡。'
>>> len(s)
8
>>> b = s.encode('utf-8')
>>> len(b)
24
>>> 

Here we are counting 8 characters/graphemes, even though it encodes to 24 utf-8 bytes.

[kelson42]

@audiodude What is the result with the string of openzim/mwoffliner#2022 (comment) ?

[audiodude]

$ python
Python 3.12.0 (main, Oct  3 2023, 17:47:52) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "विकी मेड मेडिकल इनसाइक्लोपीडिया हिंदी में"
>>> len(s)
41
>>> for i, g in enumerate(s):
...   print(i, g)
... 
0 व
1 ि
2 क
3 ी
4  
5 म
6 े
7 ड
8  
9 म
10 े
11 ड
12 ि
13 क
14 ल
15  
16 इ
17 न
18 स
19 ा
20 इ
21 क
22 ्
23 ल
24 ो
25 प
26 ी
27 ड
28 ि
29 य
30 ा
31  
32 ह
33 ि
34 ं
35 द
36 ी
37  
38 म
39 े
40 ं
>>> b = s.encode('utf-8')
>>> len(b)
113
>>> 

Yes I see the problem now. Multiple unicode code points/code units/"characters" are combined to create one grapheme.

[audiodude]

I still think that the openzim docs should contain a better explanation/example with information from these discussions, but that's a fight for another day.