Implement website language switching mechanism

[andrewdavidwong]

[Branched from #2652.]

Implement a language switching mechanism on the website (highly non-trivial, since we can't use Jekyll plugins).

[andrewdavidwong]

See @tokideveloper's comment: #2652 (comment).

[tokideveloper]

My proposed poor man solution (#2652 (comment)) has some advantages and disadvantages and there are many ways how to implement it. Here my thoughts (no indentation means new thought; indentation means related thought; bold means solution):

• All Markdown files have to be updated each time a new target language is listed on Transifex, since the language switch has to be extended by this new language.
  • This should happen on all files at the same time, since it would be confusing to have a switch to language L2 on a page of language L1 while there is no switch back to L1 on the page of language L2.
    • Thus, this should be done by a machine.
      • Thus, we could use that machine to update the language switches in general.
• There is the question wheather the language switch shall be visible for translators on Transifex or not.
  • Mistakes could be made if a switch were visible to and editable by the translators; it could also be confusing.
    • Since we should have a machine that updates the language switches in general (see above), that machine could also insert the switches each time we get a new translated file from Transifex. This way, there is no need to show the switches to the translators.
• Now, we have the problem how to manage different versions of a Markdown file: that one with a language switch (visible to visitors) and that one without it (visible to translators, see above).
  • Way 1: There is only the online version with a language switch. The machine above should also be able to remove that language switch on the fly and upload the resulting version without the language switch to Transifex. After getting a new translated version of this file from Transifex, the machine re-inserts an (of course, up-to-date) language switch on the fly and replaces the old online version with the new one (having the language switch).
    • Since the switching mechanism should also be available on the canonical version, a contributor who wants to change the canonical version will face the language switch and could remove it or something. This way, the machine that removes the language switch could be confused, make mistakes or (in the best case) halt. This would be really bad either!
      • To solve that, the machine should remove the switch on the fly before offering the page to the contributor and re-insert it afterwards. But I don't know if this could work.
      • Also keep in mind that reinserting a switch modifies a file and, as such, could trigger the machine again, resulting in an infinite loop. Watch out for concurrent editings!
  • Way 2: There is the online version with a language switch and an offline version without the switch (doubling used space). The machine above uploads only the offline version (having no switch) to Transifex. When we get a new translated version from Transifex, the machine inserts an up-to-date language switch and replaces the matching online version with the result.
    • Now, when a contributor likes to edit an online page, he/she must be redirected to the offline version of it. After finishing editing, the machine re-inserts an up-to-date language switch and replaces the matching online version with the result.
      • This way, it should be easier to handle concurrency.
  • Way 3: There is the online version without a language switch. Each time a visitor fetches a page, the webserver (or something like that) inserts the language switch on the fly.
    • I don't know if this would be possible and practicable.
    • This is probably the way how a Jekyll plugin would work.
  • Way 4: Insert a generic JavaScript-based language switch into all Markdown files (or on the fly by the webserver) that fetches a list of available languages dynamically and provides a mechanism to change the language by downloading the desired version dynamically.
    • This would need JavaScript and thus, it would exclude cautious people who turn off JavaScript.

Feel free to discuss this or propose another suggestion.

[tokideveloper]

Currently, I dive into the Liquid template engine. Hopefully, I'll find smart way to create language switches with it.

[tokideveloper]

@andrewdavidwong:
Here is my minimum prototype implementation of a language switch plus an appropriate warning message. The code is the same for both translated and untranslated versions.

Please note, since I can't read Portuguese, I left some phrases in English.

As you might see, it works for both relative and absolute links. Using relative links, there is no need to adapt links within the documentation.

As you might see, too, the prototype can only link between doc pages. So, we'll need further work to adapt the other links on the Qubes OS homepage like the links on the top bar, especially "INTRO" and "DOWNLOADS".

What do you think about it?

[tokideveloper]

A few minutes ago, I fixed a bug in my mechanism, concerning handling the root directory. I also extended the example upto the root directory.

As you might see, too, the prototype can only link between doc pages. So, we'll need further work to adapt the other links on the Qubes OS homepage like the links on the top bar, especially "INTRO" and "DOWNLOADS".

The more I think about it the easier is that work. My linked switch mechanism works as long as every name in the root directory namespace is either

• a valid language code (de-DE, en-US, pt-BR, pt-PT etc.) or
• something else (page or directory like /doc/, /intro, / etc.) if and only if it belongs to the canonical language (en-US).

As you might see, it works for both relative and absolute links.

Okay, this is irrelevant for this issue here, since this is not part of the language switch mechanism but part of how to translate links. Edit: I think the reason for absolute paths is correct redirection etc.

Moreover, the localized warning message is a solution for #2932.

[tokideveloper]

I've revised my prototype. This includes

• adding some "docs" and comments,
• unsetting of no more used vars and
• simplifying (thus time-optimizing) the algorithm (especially when calculating the var "urlpathremainder" (former "urlremainder")).

See this version.

[tokideveloper]

Concerning the number of provided translation languages: Maybe it's beneficial if we don't show a language in the switch until a certain group of pages has been translated fully or at least sufficiently in that language, e.g. home page, intro, download, getting-started, backup etc..

The reason for this consideration is that a hopeful click to your mother tongue could lead to great disappointment if the page is not really translated yet and thus hard to understand.

Whether a language shall be listed in the switch or not can be controlled via a simple flag in a YAML file containing data about translation languages.

So, the open question is: Which pages should be translated before releasing a language in the switch?

Any other opinions or ideas?

[marmarek]

So, the open question is: Which pages should be translated before releasing a language in the switch?

I'd say main page + some basic pages reachable directly from there, for example:

• https://www.qubes-os.org/intro/
• https://www.qubes-os.org/downloads/
• https://www.qubes-os.org/doc/templates/
• https://www.qubes-os.org/doc/system-requirements/
• https://www.qubes-os.org/getting-started/

What about doc index? I guess it needs to be translated too, right? Or is it possible to extract translated titles from translated pages itself?

[tokideveloper]

What about doc index? I guess it needs to be translated too, right?

Yes, /doc/ is a normal page like the others. Since it gives a first impression of what is possible to do with Qubes OS, it should be translated early, too.

Or is it possible to extract translated titles from translated pages itself?

I had this idea, too. But even in the current version, the titles shown on the /doc/ page and the real HTML titles of the linked documents don't match in general. Also they don't match with the first headings. But it works! People don't seem to be annoyed by this. So, having a mechanism that copies the titles from the linked resources to the /doc/ page is not really necessary IMHO. A simple translation of the given link texts suffices.

[marmarek]

I'm slightly worried about people assuming translation status based on page title being translated in the index. It can be misleading in both ways (having untranslated title for translated page and translated title for untranslated page).

[tokideveloper]

I'm slightly worried about people assuming translation status based on page title being translated in the index. It can be misleading in both ways (having untranslated title for translated page and translated title for untranslated page).

Okay, I see. Just to make sure: Do you mean with "title" an item string on the doc index page?

If yes, we could add a translation convention especially for the doc index:

• Only translate an item of the doc index if and only if the related page has been translated fully or almost fully.

[tokideveloper]

So, the open question is: Which pages should be translated before releasing a language in the switch?

I would add the support page and the FAQ page since they might be helpful for the user when experiencing problems:

• https://www.qubes-os.org/support/
• https://www.qubes-os.org/faq/

I would also add the pages all translators will probably visit:

• https://www.qubes-os.org/doc/contributing/
• https://www.qubes-os.org/doc/doc-guidelines/
• https://www.qubes-os.org/doc/glossary/
• page(s) concerning translation workflow

The reasons are similar to the reasons why Assembly languages were invented: They're easier to write and much less error-prone than machine code languages. It's like making translation processes self-reinforcing.

[andrewdavidwong]

FYI, for #5308, we're planning to add release numbers to documentation page URLs, e.g.:

https://www.qubes-os.org/doc/r4/installation-guide/

Since the localization effort is concerned with URLs, I want to make sure we coordinate this change with you. Do you foresee any problems?

[tokideveloper]

@andrewdavidwong commented on 10. Sept. 2019, 05:58 MESZ:

FYI, for #5308, we're planning to add release numbers to documentation page URLs, e.g.:

https://www.qubes-os.org/doc/r4/installation-guide/

Since the localization effort is concerned with URLs, I want to make sure we coordinate this change with you. Do you foresee any problems?

Thank you for involving us! Here are my questions:

What will the permalink definitions in _config.yml look like?

In this post, I proposed to collect all permalinks (defined in the YAML front matters) used by the website. Is there a way to do so without the need for permalinks in YAML front matters? For example, is there a way to find out the URL paths of the last Jekyll output? (I just quickly consulted the internet unsuccessfully.)

The language switch in its most recent proposed version assumes that each URL path of an original MD file and the URL path of its translated file will only differ in their prefix, which is a language code only in case of the translated file. So, if we have the possibility and permission to specify the adapted permalinks in the translated files (which is breaking the rule number 1, in a sense) then it's fine. (Redirects etc. should also be adapted.)

Since you plan to redefine the URL paths in the website repos anyway: What about adding a literal /en-US prefix to the general permalink definitions for reserving space for a language specifier? (just a suggestion)

[andrewdavidwong]

What will the permalink definitions in _config.yml look like?

It's in flux, but I'm currently thinking:

collections:
  doc:
    output: true
    permalink: /doc/:path/

with release-specified subdirectories, which would result in URLs like this:

https://www.qubes-os.org/doc/r4/templates/

In this post, I proposed to collect all permalinks (defined in the YAML front matters) used by the website.
Is there a way to do so without the need for permalinks in YAML front matters?

In principle, yes, using Jekyll variables, e.g., page.url, and iterating over all pages. In practice, I haven't been able to get this to work yet. (However, generating the URLs mentioned above without specifying any permalink in the YAML header of each page works.) It seems that we would need to use a _pages directory, similar to _posts, which conflicts with our _doc submodule setup.

Since you plan to redefine the URL paths in the website repos anyway: What about adding a literal /en-US prefix to the general permalink definitions for reserving space for a language specifier? (just a suggestion)

That is possible. However, it would be preferable to add those just in the _config.yml of translated repos, e.g.:

# Project Settings
project_url: "https://www.qubes-os.org/pl-PL"

which would result in URLs like this for all pages (not just doc pages):

https://www.qubes-os.org/pl-PL/doc/r4/templates/