Bringing translations to the next level.

[machour]

Dears,

As mentionned in a previous issue, and on gitter, I think that the current system used for handling translations is flawed.

Here are the major problems I've seen/I predict:

• keywords doesn't often make a lot of sense (had to check the source code to guess the context/meaning of some of them)
• contributing becomes "harder". I'm introducing a new feature with some sentences, I have to check : If these sentences exists as keywords somewhere. If not, where should I put them ? under what namespaces ?
• even reading the code source gets tedious: https://github.com/gitpoint/git-point/blob/master/src/auth/screens/events.screen.js#L491-L502
• concatenation is bad: user + ' made ' + repository + 'public', is perfect for English, but it may not be suited for another language. Let's not speak about the mess it will be when we introduce a first RTL language
• keeping translations up to date is tedious, as if someone changes the english value of the key, there's a big chance it will be missed by translators : c5b90da
• currently, adding a entry en.js break all other languages. Contributors are using Google Translate in order to avoid this: feat(starred-count): show starred count #271 (comment)
• last but not least, making sure that all translations are up to date before pushing next release to App stores is impossible for @housseindjirdeh and will require him to wait for all translators to respond.

In the past days, and with the help of @Antoine38660 (<3 buddy) I worked on a proof of concept for a better translation system, designed to solve all these problems and make contributors lifes easier.

Its concept is clearly inspired a lot by a PHP framework called Yii, which handles i18n in a clever way. [1]

The POC is available here: https://github.com/machour/rn-i18n-poc

(please keep in mind, I only did jQuery in the past and discovered const, import, yarn, let, export a week ago. Antoine did its best to enhance stuff, but I kept on breaking things.)

Here are the main features of this poorly coded idea :

Coding your app

• The application is developed using english sentences, surrounded by a function, and using placeholders to inject values :

t("{user} made {repository} public", { 
	user: "Houssein",
	repository: "gitpoint/git-point"
});

• Injected values can be strings, or ... React Components !

  t("{user} made {repository} public", { 
    user: "Houssein",
    repository: (<Button title="gitpoint/gitpoint" onPress={()=>console.log("you clicked")}>gitpoint/git-point</Button>)
  })
  

• Pluralization handling :

t("There {n,plural,=0{are no cats} =1{is one cat} =*{are # cats}}!" {
 n: 0
}) // There are no cats!

t("There {n,plural,=0{are no cats} =1{is one cat} =*{are # cats}}!", {
 n: 1
}) // There is one cat!

t("There {n,plural,=0{are no cats} =1{is one cat} =*{are # cats}}!", {
 n: 42
}) // There are 42 cats!

You can see more features by running the app from the repo.

Actual translations

A string extraction script is provided and takes the following parameters :

• sourcePath: The directory to be scanned for strings
• translator: The function used in your app to trigger translations (t in the above examples)
• languages: A comma separated list of languages code
• messagePath: The directory where the messages files will be generated

The script produces a JS file with this structure (fr.js for example)

export default {
  "I think that {user} made {repository} public": "Je crois que {user} a rendu {repository} public",
  "This string is not translated and will automatically fallback to {language}": "",
  "This string should not trigger placeholders processing": "Cette phrase ne doit pas invoquer le processing des remplacements",
  "This string uses {nested}": "Cette phrase utilises des {nested}",
  "nested translating!": "traductions imbriquées :)",
  "Oops, I forgot to pass my {placeholder}": "Oups, j'ai oublié de passer mon {placeholder}",
  "A simple sentence without placeholders": "Une phrase simple sans motifs de remplacements",
  "Two {consecutive} {placeholders}": "Deux {placeholders} {consecutive}",
  "consecutive": "consecutifs",
  "placeholders": "motifs de remplacements",
  "A sentence {0} using {1} numerical {0} placeholders": "Une phrase {0} utilisant {1} des motifs {0} numériques",
  "A simple sentence with only one placeholder passed as a {0} without wrapping it in an array": "Une phrase simple avec un seul motif de remplacement passé en tant que {0} sans avoir à en faire un tableau",
  "There {n,plural,=0{are no cats} =1{is one cat} =*{are # cats}}!": "Il y a {n,plural,=0{aucun chat} =1{un chat} =*{# chats}}!",
  "Hello {name}. This is a component using t()": "Bonjour {name}. Ceci est un composant utilisant t()"
};

• Any string not translated ( = "" ) will fallback to english (or the native language used to develop the app)
• The script takes care of merging new extracted sentences with previously translated ones
• If a sentence is removed from the application, it will be surrounded with @ symbols to hint the translator.

An additional script could be developed to make sure that all translations are up-to-date :

1. run the extraction process
2. iterate on generated files to test : if there are missing translations (empty) or obsolete ones (@ some old string@)

Having a running POC to show, and not so many skills, I decided to stop here, show you what I've been rambling about lately and have a first round of comments (or applauses 😆 ).

Could that be the next great way of handling translations in a React Native app ? (and I know, it could be made into a framework agnostic base, with several implementations).

Comments please.

[1] - If you want to see where I stoleborrowed the concept : http://www.yiiframework.com/doc-2.0/guide-tutorial-i18n.html

[housseindjirdeh]

This is fascinating :O. I'm so surprised I haven't heard of Yii before!

Some thoughts:

1. After pulling down your POC repo (thanks for setting that up btw), it looks like the English copy is the key here for the translations files correct? I actually really like that. It makes being able to digest translated strings in components so much easier without having to check the specific translation files. My one concern with this however is if I want to change the English copy slightly. For example:

{ t("A simple sentence without placeholders") } to { t(Super simple sentence without placeholders") }

I'm assuming translations will fail unless I update the key (which is the string here) in each of the translation files correct? I'm only thinking it may be slightly easier to make this mistake with long key names instead of short distinct ones but I really don't think that's a deal breaker.

2. Not sure how the script works. Why is there a script to generate the localization files if we can then modify it? I still don't understand where exactly you write the translations for fr.js if the script auto generates this file :think:

3. With pluralization, I'm assuming the idea is we pass in a component variable to make rendering different text easier correct? For example:

 t("There {n,plural,=0{are no products} =1{is one product} =*{are # produts}}!", {
  n: products.length
 })}

I think that is just amazing 😍

If a sentence is removed from the application, it will be surrounded with @ symbols to hint the translator.

WOWOWOWOW THIS IS A GAME CHANGER. Do we need to run the script in order to have this show??

Injected values can be strings, or ... React Components !

That is awesome, but just thinking out loud here 🤔 Isn't there an issue with having components rendering between text when it comes to styling? I guess that's up to the user correct?

---

Some closing thoughts:

This looks powerful. Although I'm still trying to wrap my head around how all of it works, I can already see features that may blow other super simple key:value translation solutions out of the water. I think this can definitely be used for more than React Native and I'm happy to have GitPoint be a pilot ground for this library. @machour @Antoine38660 : you guys are wizards!

[machour]

So thrilled you liked the idea 🙌 🙌 🙌

1. Exactly, the English (source) sentence is the key in this system. If you do change an English sentence, translators will have to catch up. Until that, the translated app will display the english sentence, as the translations are outdated.

In fact, I consider this a good feature, as the update you made changes the meaning of the sentence, and translations have to be updated accordingly.

2. Think of the script as a helper, automatically doing things you'd be doing manually.

I still don't understand where exactly you write the translations for fr.js if the script auto generates this file

I put them in that exact same file, as the extraction process won't override them ;)
Here's how the script works :

• it scans the source code and extracts all sentences meant to be translated
• it loads all the existing translations files, and read the translations
• it performs a smart merge and writes back the file

The merge rules are as follow :

• if a new sentence is found in the source code: add it with an empty translation
• if a sentence is found in the source code and was already translated: keep its translation
• if a sentence is found in the translation file, but not in the source code: this sentence probably existed in the source code at some point, then was removed. There is no point in keeping it in the translation file, so we signal it to translators by surrounding it by @ marks.

Let's take your change in the 1st point as an example :
Originally we'd have in fr.js:

"A simple sentence without placeholders": "Une simple phrase sans motifs de remplacement",

One you've updated the source code with your change, and re-run the script, we'd have :

"Super simple sentence without placeholders": "",
"A simple sentence without placeholders": "@Une simple phrase sans motifs de remplacement@",

As a translator, keeping the obsolete sentence helps me to copy paste parts of it in the new sentence, before deleting it manually:

"Super simple sentence without placeholders": "Phrase super simple sans motifs de remplacement",

3. That's exactly it. It helps avoid cluttering the source code with if/else to get the plural form right. This is even helpful for English !

Yii also supports the 'select' keyword, which would give us something like this:

t('{name} is a {gender} and {gender,select,female{she} male{he} other{it}} loves Yii!', {
    'name' : 'Snoopy',
    'gender': 'dog',
}); // Snoopy is a dog and it loves Yii!

there is a lot more syntax sugar (numerical placeholders, single placeholder passed as a string, ..), all meant to simplify the i18n task to the max.

4. Explained in point 2.

Now you can easily imagine a pre-release yarn task that would run the extraction script on all languages, then the generated/merged files for empty or obsolete translations. 😍

5. I didn't really have styling in mind when developing this part. I was just thinking about @Antoine38660 next PR for repository linking and how could a React Component/complex object be passed as a placeholder.

For now the translated sentences are always wrapped in <Text> by default, and in a <View> if a component is detected in a placeholder. There are cases where we don't want any wrapping at all. Others where we prefer something else than View as a wrapper.

Maybe a third optional parameter could do that, or maybe this needs a lot of more thinking, especially if we're going framework agnostic.

[housseindjirdeh]

1. if a new sentence is found in the source code: add it with an empty translation
2. if a sentence is found in the source code and was already translated: keep its translation
3. if a sentence is found in the translation file, but not in the source code: this sentence probably existed in the source code at some point, then was removed. There is no point in keeping it in the translation file, so we signal it to translators by surrounding it by @ marks.

Oh my god. This is a game changer. I can't even explain how much of an improvement this can be over the majority of existing translation libraries. I can almost guarantee that the number of people who would flock to use a library like this would be insane. Framework agnostic (in my opinion) would be something that would make this probably the best JavaScript translation library out there.

Now you can easily imagine a pre-release yarn task that would run the extraction script on all languages, then the generated/merged files for empty or obsolete translations.

😍 😍 😍 Throwing ideas out here, but what if you could output consoles to the terminal (and/or CI) so when this step is run as part of the build process: all the empty and obsolete translations are shown to the user 😍 😍 😍 .

For now the translated sentences are always wrapped in by default, and in a if a component is detected in a placeholder. There are cases where we don't want any wrapping at all. Others where we prefer something else than View as a wrapper.

Maybe a third optional parameter could do that, or maybe this needs a lot of more thinking, especially if we're going framework agnostic.

This will definitely need some more thinking in my opinion but I don't think this is critical at all. Happy to have this experimentally with our app since it might work in some contexts such as @Antoine38660's.

---

Okay so I'm excited, and I'm sold. How soon are you envisioning releasing your library? More than willing to have this included into GitPoint as it's first app, after you cleared up some points I can already see how much this can help!

[housseindjirdeh]

@RolfKoenders @andrewda @alejandronanez @lex111 etc... don't hesitate to drop some tips/suggestions/concerns if you have any at all as well!

[andrewda]

Oh my gosh this is awesome. I'd love to contribute!!

[alejandronanez]

Damn this is sweeet. Will check tomorrow morning 😍

[housseindjirdeh]

Echoing @andrewda, will be happy to contribute so let us know if you need help with anything! @machour

[Antoine38660]

I'm happy to know that you think the idea is cool and can be used for Gitpoint and other projects!

Our major problem was (is) to separate each part of our code / repository to create a lib "distributable"

[machour]

😊 😊 😊 , thank you all for the positive feedback !

As for the release date, I don't have any in mind, but I think that we could come really fast with a first solid beta.

@Antoine38660 and I seems to lack the technical background in JS on how to implement this in the best manner, so I can't really come with a first version for us to work on. Thus I'd really need your help to be set on the right track and do a collective kick-ass i18n library.

I just compiled this ticket into a new document on my poc repository :
https://github.com/machour/rn-i18n-poc/blob/discussion/Discussion.md
This will allow us to discuss the development furthermore by adding comments on this PR and keep the conversation organized :
https://github.com/machour/rn-i18n-poc/pull/1/files

Closing this issue and moving there to add some comments ;)

[alejandronanez]

@machour Just for reference, did you check this repo already? Maybe you can do something like them.
https://github.com/i18next/react-i18next

[machour]

@alejandronanez oh, nice catch! i18next seems to be a great library and we sure may inspire stuff from them.

[lex111]

Ooooh, as usual, I missed an interesting discussion 😞 . So many details, I'll just write that I like the idea! 👍 I propose to consider the existing solutions:
https://github.com/yahoo/react-intl
https://github.com/joshswan/react-native-globalize

Will they help us?

[RolfKoenders]

I only read halve of the discussion but already wanted to share that this is amazing!