Syntax Simplicity

[nbouvrette]

Is your feature request related to a problem? Please describe.
Linguistic challenges are complex and having a simple way to solve them can also be complex. Message Format seems to have kept a certain level of simplicity which makes adoption easier, especially by non-technical users like linguists.

Describe the solution you'd like
I would like for the new syntax to remain simple (at least as simple as Message Format today, or even more simple if this is even possible).

Describe why your solution should shape the standard
By having a simple syntax, it will help both authors and linguists to manipulate raw syntax without having to spend too much time learning it.

There is also a limit in terms of complexity that linguists will be willing to learn, especially if we are aiming for global adoption. Linguists are language experts, not engineers.

If we presume that raw syntax cannot be translated directly by linguists without the need for tools, this means that we will have to rely on other ways to get the translation done.

If the raw syntax is too complex and that we have to support some sort of "linguist friendly format", I am not too sure how this will work for some inflection problems (e.g. adding language-specific syntax by language specialists).

Additional context or examples
Based on personal experience I have seen linguists directly edit several existing syntaxes such as:

• ChoiceFormat: quick learning curve. Has been used for a while in legacy projects to help solve some linguistic issues like plurals
• MessageFormat: can ramp up linguists in about 1 hour and received very good feedback when solving plural issues

[mihnita]

Just a note: linguists don't handle the syntax directly, they use dedicated tools.
So it does not matter from that side.

We should be able to map our data model (independent of syntax) to the most common data model supported by localization tools.

[nbouvrette]

Just a note: linguists don't handle the syntax directly, they use dedicated tools.
So it does not matter from that side.

I disagree with this and I think it would be a good topic so see how others are translating MessageFormat today.

From my experience, we decided to train linguists directly with the raw syntax and it worked really well - the main reasons for this:

• Tools are insufficient - poor integration with most major commercial TMSes
  • I don't think that the major TMS Company (SDL) supports this well - we could double-check if anyone has more info there. I'm pretty sure a lot of TMS also do not provide native support yet
  • Because the syntax does not enforce full sentences, there are a lot of nested syntax that can break most native TMS support
  • Even if the TMS integration works, the way messages are fragmented can make it really hard to translate (again because full sentences are not mandatory)
• Fluency: if you want to use explicit rules in plural statements, this might not apply to all languages and TMS does a poor job with this. This might be another topic on the other hand - maybe we would want to disable explicit rules? (see related thread)

Keep in mind that MessageFormat has been there for a while and is still in a state that I would consider inadequate in terms of tooling (XLIFF is similar).

The localization industry moves slowly, so if we expect to have tools to use the new syntax, we know that adoption will most likely also be slow.

I would prefer simple, linguist friendly syntax :)

[mihnita]

XLIFF has a lot more adoption than MessageFormat, which is not as used, even if it was around for a long time.

The adoption is not about syntax.

---

The bigger difficulty is the data model.
If that is too complex and has no decent mapping to the way existing localization chains represent the data then it will not be supported.
Writing an import / export filter is easy, if the data models are close enough.
But if they are too far away it would require architectural changes in the way everything else works (translation memories, leveraging, validation, UI, etc.)

This is the same as creating COBOL to sound very English-like in the hope that programming can be done by business people. Programming is hard because it is hard, not because the program does not read like English.

Linguistically correct handling of translation is hard because human languages are messy.

---

The second part of the (non)-adoption is about politics.
XLIFF was created to allow for localization vendor / tooling independence.
So the tool vendors that had de-facto monopoly had no interest to implement it properly.
I know of one of the major tool that had (has?) it's own extension to represent basic stuff like comments, even if xliff has a standard way to do it.
Major clients (without naming names) didn't help either, by creating their own tools that supported some of the advanced features they wanted. And that created a different kind of lock-in.

---

This is why I am (sometimes a bit too?) aggressive in rejecting complicated features that will prevent adoption.

Saying "translators should be able to handle it directly" means we give up on adoption from the get go. We design it so it will not be adopted.

[nbouvrette]

The adoption is not about syntax.

I agree it's not all about the syntax - I think its a combination of the followings:

• Easy of use (typically simplicity helps which is the proposal here)
• Good visibility online (aka marketing) - this includes demos tools like Fluent Playground
• Good integration with existing tools - considering all programming languages and TMSes (I think flexibility is key)

Writing an import/export filter is easy, if the data models are close enough.

This is the part I'm not sure I can picture easily? We didn't talk too much about continuous localization yet, but I think that related to file formats if we keep the current state that MF supports, which from my experience is both simple in terms of syntax but also file format agnostic in terms of storing the syntax in keys - this should align well with the simplicity side of the solution.

If you think that to adopt the new syntax we will need to have all users implement some sort of import/export filter, then I would be curious to understand where this fits in a continuous localization pipeline? And also why this is needed because it does increase complexity.

The second part of the (non)-adoption is about politics.

I think we have to be careful with the word "politics" - it can easily be used the wrong way :-)

So the tool vendors that had de-facto monopoly had no interest to implement it properly.

That's my point - if vendors are not interested, then what? businesses use vendors... so for a new solution it is a dependancy - I think it is important to either:

1. Make it interesting enough for vendors to implement at scale (very challenging)
2. Or make it simple enough to not require much from the vendors (this is what I'm proposing)

Major clients (without naming names) didn't help either, by creating their own tools that supported some of the advanced features they wanted. And that created a different kind of lock-in.

Yes, this is a reality of the localization landscape, which I think is mostly due to the vast complexity of languages and the lack of a solution that covers all needs.

Saying "translators should be able to handle it directly" means we give up on adoption from the get go. We design it so it will not be adopted.

I'm really not sure I understand what you mean here, are you saying we cannot keep the syntax as simple as it is today with MF? Because if we can, my experience is that this will help adoption, not the opposite.

If we can't, then maybe you are seeing something I am missing?

[jamuhl]

XLIFF was created to allow for localization vendor / tooling independence.

Did it achieve that?

Isn't xliff more a catalog format than a syntax? Might be not really understand why xliff is that important beside trying to be a interchange format...

At locize we support the most basic stuff out of XLIFF - to import/export -> but most time the files we get from other vendors are zero compatible with our tooling...our approach is rather different to the old ones -> keeping things simple and close to the runtime format to enable a real continuous localization (publishing usable runtime translation files to CDN)

[Fleker]

Tools are insufficient - poor integration with most major commercial TMSes

I think the poor tooling is a separate issue. While I do agree having linguists learn the syntax can be beneficial, I wouldn't want it to be a primary goal of the format. There should be better tooling in general.

[mihnita]

Continuous localization as nothing to do with adoption.

XLIFF was created to allow for localization vendor / tooling independence.

Did it achieve that?

Partially. If you choose the right subset of the XLIFF features then yes, it is supported.

Isn't xliff more a catalog format than a syntax?
Might be not really understand why xliff is that important beside trying to be a interchange format...

I see it as more than a catalog.
There is structure there. There are groups, text units, segments, placeholders.
There is info about placeholder types (open / close / standalone), flags for the placeholders, and ways to map from format-specific placeholders to the xliff (language independent) ones.

With proper XLIFF support one would be able to translate ...{user}... and leverage ...%s... and ...$user... and ...{0}... without damaging anythging.
Would be able to leverage between formatted content. etc.

In a way I think xliff 1.2 was too ambitious.

[mihnita]

Here is a pretty good document on XLIFF adoption:
https://www.localizationworld.com/lwdub2014/feisgiltt/slides/Chase_Keynote.pdf

It touches adoption, feature creep, and more.

[mihnita]

Writing an import/export filter is easy, if the data models are close enough.

This is the part I'm not sure I can picture easily?

This is about the way current localization tools represent data.

They are very much geared on a 1:1 mapping.
I'll list here some of the assumptions the current workflow make (partial list):

• You give me an English string, I give you a French one with the same meaning
• If the English has 2 placeholders, the French string will have the same 2 placeholders (maybe in a different order)
• Translators are not allowed to modify any kind of code (placeholders or anything else), then can only move things around, if needed.
• Strings can be translated independently. I get a message today, I translate it and send it back. It does not matter what was translated yesterday, and I can't even access it. So I can't go back to the company name translated 2 weeks ago and add a genitive form

Anything outside this view of the world breaks functionality.
For example plurals.

The source (English) is:

• you deleted the last file
• you deleted {count} files

Russian needs to send back 4 messages. Things break.

OK, "flatten" all messages into 1 and allow translators to edit things directly (your solution, and Fluent, for more complex messages)
The side effect: the rest of the tooling stops working properly.

• Validation will complain, because the source had one placeholder ({count}), but the translation has 3 (the few and many will add 2 more)
• Some validation will complain because the target is way too long (rules like "length(target) < 1.5 * length(source)")
• Leveraging will break, especially advanced features like fuzzy matching, context (previous + following message)
• Spell checkers will "choke" on pieces of code, will complain that you did not translate stuff like "year: numeric"
• Consistency checking and glossary validation will give false negatives
  etc.

These are data model incompatibilities.
Are a lot harder to change, and are not solved with a simple filter, or "let translators change the string as they see fit"

[mihnita]

Or make it simple enough to not require much from the vendors (this is what I'm proposing)

This breaks a lot of the useful features the vendors built into their tooling.

It is like saying "make things simple enough so that developers don't need an IDE, they can use a simple text editor" and "a linter / compiler will tell them if there are problems"

So now they lost not only syntax highlighting, but refactoring, and suggestions ("intellisense" or whatever you want to call it) and more.
(yes, translators also do "refactoring" of sort, with global terminology updates)

This just pushes back the CAT tools to the "dumb text editors" stage.

That also means lower linguistic quality (forcing translators to focus on technical aspects).
And slows them down. A lot.
In a world where translators are payed by (source) word count this matters to them. A lot.

Imagine someone tells you "starting tomorrow you can't touch Eclipse / IntelliJ / Visual Studio, you develop software in Notepad"

[zbraniecki]

Imagine someone tells you "starting tomorrow you can't touch Eclipse / IntelliJ / Visual Studio, you develop software in Notepad"

I agree that we should leverage power of CAT tools, but if we're designing for the Open Web, and something that we aim to suggest for a standard, then the reverse of your claim is also true:

Imagine if someone told you that in order to write in a new programming language for the Web you need to use Eclipse, because there's no way for you to work with it in anything else.

The principle of least power is described by W3C - https://www.w3.org/2001/tag/doc/leastPower-2006-01-23.html - and I believe we should aim for our system to work with notepad.

Then, if you plug some tooling, it should work better.

And if you use a CAT tool it should work amazing. And we should make it easy to develop CAT tool integrations, command line tooling, etc. But I'd like to avoid a future in which our outcome is basically unreadable/unwritable without sophisticated CAT tools.
I don't think anyone is advocating for it, but in the push for "CAT tools will fix X" we may end up there non the less if we're not careful.

[mihnita]

I think I did not touch all points:

Good integration with existing tools - considering all programming languages and TMSes (I think flexibility is key)

This is where there is friction. Programming languages are flexible, and value flexibility. TMSess are a lot less flexible.

If you think that to adopt the new syntax we will need to have all users implement some sort of import/export filter, then I would be curious to understand where this fits in a continuous localization pipeline? And also why this is needed because it does increase complexity.

This I don't understand.
There will be a filter, not matter what.
Any tool we use (unless is a 100% dumb text editor) needs to "understand" a syntax, and convert from a file format to an internal, in memory representation.
Unless one implements a tool for our format ONLY, and the internal data structures map exactly to our data model, then they will need a filter.

It is like asking "why an image editor needs a filter for GIF files".
Unless the internal model assumes palette based 256 colors, transparency is on / off (not degrees of transparency), then you need a filter.
I see not connection between continuous localization and the need for a filter, and I don't see how having one makes thing more complicated.

I'm really not sure I understand what you mean here, are you saying we cannot keep the syntax as simple as it is today with MF? Because if we can, my experience is that this will help adoption, not the opposite.

The current MF syntax is not well supported. Got some adoption, but not really enough, for something that was around for about 15 years. Or, got developer adoption, but not TMS support.

And yes, I say that we can't keep the syntax as simple as it is today.
Because we want to add TONS of extra new features.
Inflections are hard, and will make things harder.
Again, nothing to do with the syntax. They are hard because human languages are hard.