Support splitting the PO files into smaller units

[mgeisler]

We currently work with monolithic file: mdbook-xgettext writes all messages to a single POT file (typically messages.pot) and mdbook-gettext read a single PO file as well. We've seen in Comprehensive Rust how these files can grow very large: the PO files are around 800 kB and 24k lines in size.

It would be interesting to support splitting the POT file into smaller units. We could let the book outline decide the splitting and split on each top-level chapter. More elegantly, we could have a splitting depth: the current behavior would be splitting at depth 0, depth 1 would split at top-level chapters, depth 2 would go one level further, etc.

This could in principle be implemented today by scripting calls to msggrep --location (to split the POT file into smaller units) and msgcat (to compile multiple PO files). However, a native solution will likely be cleaner overall. The splitting with msggrep and combining with msgcat suggests an implementation strategy: keep everything like today, and split the Catalog into units before saving them in mdbook-xgettext. Similarly, create a large Catalog from many smaller ones in mdbook-gettext.

[antoniolinhart]

Hey,

This bug looks interesting! Would I be able to pick it up?

Is Catalog a data type for mdbook or just an unofficial term for a POT file? I tried to look up Catalog data type but I didn't see it associated with mdbook.

Thank you!

[mgeisler]

Hi @antoniolinhart, I'm glad you're interested in it!

The Catalog type is from polib, the library we use to read/write the PO files.

[antoniolinhart]

Would I please be able to added as a contributor to this repo so that I can create a branch and push code?

I started a discussion in #73, but want to do some experimenting to see if I can test my changes in comprehensive-rust by setting the mdbook-i18n-helpers dependency on my specific commit in this repo.

Thanks!

[mgeisler]

Would I please be able to added as a contributor to this repo so that I can create a branch and push code?

The recommended workflow would be to create a fork on GitHub and push to a branch in that repository. That will be your own playground which you have complete control over. You can create and delete branches in your fork as you like, without any danger of making permanent changes in this repository.

I started a discussion in #73, but want to do some experimenting to see if I can test my changes in comprehensive-rust by setting the mdbook-i18n-helpers dependency on my specific commit in this repo.

Since the coupling between Comprehensive Rust and this repository is via binaries in your $PATH, you will actually be able to test this fully locally by something like

cd your-clone-of-mdbook-i18n-helpers
cargo install --path .
cd your-clone-of-comprehensive-rust
MDBOOK_OUTPUT='{"xgettext": {"pot-file": "messages.pot"}}' mdbook build -d po

The last command there will find the mdbook-xgettext binary which you just installed with cargo install.

[antoniolinhart]

The recommended workflow would be to create a fork on GitHub and push to a branch in that repository. That will be your own playground which you have complete control over. You can create and delete branches in your fork as you like, without any danger of making permanent changes in this repository.

Wow! This makes a lot more sense now... I will do this for my next PRs. The process is easier than I thought. Thanks!!

[mgeisler]

Wow! This makes a lot more sense now... I will do this for my next PRs. The process is easier than I thought. Thanks!!

You can definitely get write access to this repository as well — I just wanted to point out that it's not strictly necessary most of the time 🙂 Your PRs will automatically start running tests after the first one is merged.

[mgeisler]

Hi @antoniolinhart, there is a little thing which we will need to tackle down the road: linking to the correct PO file when there are multiple of them.

Right now, we create a link to the xx.po file in our JavaScript: index.hbs. This doesn't work super well because the files are so gigantic.

When we generate smaller files, we will need a way to know the correct file for each page. @sakex is working on adding support for better templating in #70, and with that, we should be able to do more complex string manipulation in the templates — but we still need a mapping of Markdown page to PO file somehow. That could perhaps be emitted by mdbook-xgettext (write a TOML or JSON file to a known place, read this in the template).

Finally, we have google/comprehensive-rust#1075, which is also slightly related. I think we could mark that as closed if we get much smaller PO files in general, but perhaps there is a way to emit all the necessary information?

[antoniolinhart]

Hi @antoniolinhart, there is a little thing which we will need to tackle down the road: linking to the correct PO file when there are multiple of them.

Right now, we create a link to the xx.po file in our JavaScript: index.hbs. This doesn't work super well because the files are so gigantic.

When we generate smaller files, we will need a way to know the correct file for each page. @sakex is working on adding support for better templating in #70, and with that, we should be able to do more complex string manipulation in the templates — but we still need a mapping of Markdown page to PO file somehow. That could perhaps be emitted by mdbook-xgettext (write a TOML or JSON file to a known place, read this in the template).

Outputting a map of a markdown page to a PO file should be doable. My implementation should be easy to modify to carry through the path of the MD file so we can use it and place it in the map!

Finally, we have google/comprehensive-rust#1075, which is also slightly related. I think we could mark that as closed if we get much smaller PO files in general, but perhaps there is a way to emit all the necessary information?

Yeah, I think having the above output file will help a decent bit with this, we'll just need a way to find the line number in the po file. I'm not fully familiar with the options here, where would that logic go? Is an option to output a second file (from xgettext) that has a line number for each message in the specified PO file?

[mgeisler]

I'm not fully familiar with the options here, where would that logic go? Is an option to output a second file (from xgettext) that has a line number for each message in the specified PO file?

You actually just need one line number per Markdown file. So that when someone clicks the edit link in the top right, we can link to the correct line. Go ahead and try it yourself on both the English version and a translated version — nothing bad will happen, you'll just open an editor which you can close again.

We can output more files if that helps with looking things up at publication time or runtime. My hope will be that the new template system can process such a file when we run mdbook build — you should coordinate this with @sakex.