POT Generator: Add support for TRANSLATORS: and NO_TRANSLATE comments

[dalexeev]

• Closes Add support for generating comments from tr() and tr_n() functions for gettext (pot) translation system  godot-proposals#8555.
• Closes Allow EditorTranslationParserPlugins to provide comments for POT generation godot-proposals#8917.
• Closes Allow skipping script lines from POT generation godot-proposals#10590.
• Supersedes Allow EditorTranslationParser to add POT comments #87401.

Example

extends Node

func _ready() -> void:
    print(tr("tr1"))
    print(tr("tr2", "ctx_a"))
    print(tr("tr3")) # NO_TRANSLATE
    print(tr("tr4")) # TRANSLATORS: Message 4
    # TRANSLATORS: Message 5
    print(tr("tr5"))
    # Above comment.
    # TRANSLATORS: Message 6 Line 1
    # Message 6 Line 2
    # Message 6 Line 3
    print(tr("tr6"))
    # Above comment.
    # TRANSLATORS: Message 7 Line 1
    # Message 7 Line 2
    # Message 7 Line 3
    print(tr_n("tr_n7", "tr_n7_plural", 0, "ctx_b"))

    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.1
    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.1
    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.2
    print(tr("tr8", "ctx_d")) # TRANSLATORS: Message 8.3

    # TRANSLATORS: Message 9

    # Empty line ^^
    print(tr("tr9"))

    print(tr("tr10")) # TRANSLATORS: Message 10
    print(tr("tr11"))

# LANGUAGE translation for temp for the following files:
# res://node.gd
#
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: temp\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8-bit\n"

#: node.gd
msgid "tr1"
msgstr ""

#: node.gd
msgctxt "ctx_a"
msgid "tr2"
msgstr ""

#. TRANSLATORS: Message 4
#: node.gd
msgid "tr4"
msgstr ""

#. TRANSLATORS: Message 5
#: node.gd
msgid "tr5"
msgstr ""

#. TRANSLATORS: Message 6 Line 1
#. Message 6 Line 2
#. Message 6 Line 3
#: node.gd
msgid "tr6"
msgstr ""

#. TRANSLATORS: Message 7 Line 1
#. Message 7 Line 2
#. Message 7 Line 3
#: node.gd
msgctxt "ctx_b"
msgid "tr_n7"
msgid_plural "tr_n7_plural"
msgstr[0] ""
msgstr[1] ""

#. TRANSLATORS: Message 8.1
#. Message 8.2
#: node.gd
msgctxt "ctx_c"
msgid "tr8"
msgstr ""

#. TRANSLATORS: Message 8.3
#: node.gd
msgctxt "ctx_d"
msgid "tr8"
msgstr ""

#: node.gd
msgid "tr9"
msgstr ""

#. TRANSLATORS: Message 10
#: node.gd
msgid "tr10"
msgstr ""

#: node.gd
msgid "tr11"
msgstr ""

[AdriaandeJongh]

Just curious, because it's missing from the examples: what happens to comments surrounding a single line with 2 tr()'s? Eg:

# TRANSLATORS: hello
print(tr("key_here"), tr("another_key"))

And when there's both a comment above and behind?

# TRANSLATORS: hello
print(tr("key_here"), tr("another_key")) # TRANSLATORS: say what

[dalexeev]

what happens to comments surrounding a single line with 2 tr()'s? Eg:

# TRANSLATORS: hello
print(tr("key_here"), tr("another_key"))

This will be applied to both strings, the parser does not check if the comment is consumed. But you can write:

print(
    tr("key_here"), # TRANSLATORS: hello
    tr("another_key"), # TRANSLATORS: world
)

or:

print(
    # TRANSLATORS: hello
    tr("key_here"),
    # TRANSLATORS: world
    tr("another_key"),
)

And when there's both a comment above and behind?

# TRANSLATORS: hello
print(tr("key_here"), tr("another_key")) # TRANSLATORS: say what

Inline comment takes precedence, comment above will not be used.

[timothyqiu]

In your example,

    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.1
    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.2

generates

#. TRANSLATORS: Message 8.1
#. TRANSLATORS: Message 8.2
#: node.gd
msgctxt "ctx_c"
msgid "tr8"
msgstr ""

This makes the translators comment

Message 8.1
TRANSLATORS: Message 8.2

The expected result is

Message 8.1
Message 8.2

[dalexeev]

@timothyqiu I wasn't sure if we need remove the TRANSLATORS: prefix, so I left it like in Godot editor translations.

https://github.com/godotengine/godot-editor-l10n/blob/b8f7a2d06918f897eed69f7d481438bc48ce7595/editor/editor.pot#L1091-L1094

[timothyqiu]

    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.1
    print(tr("tr8", "ctx_c")) # TRANSLATORS: Message 8.2

@dalexeev I mean, the message tr8 effectively has is a multiline comment and the second TRANSLATORS: is currently part of the comment body (the first TRANSLATORS: can be recognized by tools like Poedit).

Perhaps the TRANSLATORS: prefix should be kept on the first line only. Maybe also generate a blank line between the two lines, since they come from different code blocks.

[dalexeev]

@timothyqiu Done, updated the example.

[Vovkiv]

Great job!
I have one suggestion, that probably should be made in separate PR, but what about adding "TRANSLATORS:" as default theme marker in editor settings?

Maybe as notice perhaps?

[dalexeev]

I think we should have a separate color for built-in comment keywords to distinguish them from user ones. I have a local branch for GDScriptSyntaxHighlighter refactoring, where I want to add highlighting of BBCode markup and special tags (@tutorial, @deprecated, @experimental) in doc comments. The same for TRANSLATORS and NO_TRANSLATE.

[timothyqiu]

Tested some common use cases locally, and it worked as expected.

[dalexeev]

Fixed a case I forgot about (non-inline comment should only be taken into account if there is no code in that line).

print(tr("tr10")) # TRANSLATORS: Message 10
print(tr("tr11"))

[molingyu]

Should we add a comment property under the Localization property group in the UI system?

[timothyqiu]

Should we add a comment property under the Localization property group in the UI system?

This falls into the same category as translation context: godotengine/godot-proposals#8061

Currently, there are several one-to-many / many-to-one situations that need to be sorted out. I don't think node's auto translate system is ready for this. At least it should be a separate PR, it's not very GDScript related :P

[molingyu]

@timothyqiu Yes, it is indeed not very relevant to this PR.

I want to reset the translation content generation process of the entire UI system, so that it can automatically collect the marked multilingual information from tscn.