Skip to content

Commit

Permalink
Merge pull request CleverRaven#64926 from jbytheway/language-stats
Browse files Browse the repository at this point in the history
Add percentage-translated statistic to language selection
  • Loading branch information
ZhilkinSerg authored Apr 12, 2023
2 parents 5434968 + 520832c commit e129921
Show file tree
Hide file tree
Showing 10 changed files with 188 additions and 70 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/pull-translations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ jobs:
env:
TX_TOKEN: ${{ secrets.TX_TOKEN }}
run: tx pull --force
- name: "Update stats"
run: ./lang/update_stats.sh
- name: Create Pull Request
uses: peter-evans/create-pull-request@v4
with:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,9 @@ tags
/lang/json/*
!/lang/json/README

# Languag translation stats intermediate files
/lang/stats/

# Netbeans IDE folders
nbproject/

Expand Down
25 changes: 21 additions & 4 deletions doc/TRANSLATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,9 +290,10 @@ translation name{ translation::plural_tag() };
jsobj.read( "name", name );
```
If neither "str_pl" nor "str_sp" is specified, the plural form defaults to the
singular form + "s". However, "str_pl" may still be needed if the unit test cannot
determine whether the correct plural form can be formed by simply appending "s".
If neither `"str_pl"` nor `"str_sp"` is specified, the plural form defaults to
the singular form + "s". However, `"str_pl"` may still be needed if the unit
test cannot determine whether the correct plural form can be formed by simply
appending "s".
You can also add comments for translators by writing it like below (the order
of the entries does not matter):
Expand Down Expand Up @@ -348,7 +349,14 @@ See the [gettext manual][6] for more information.

## Maintainers

Several steps need to be done in the correct order to correctly merge and maintain the translation files.
### Automated updates

Under normal circumstances the translation files are updated automatically by a
weekly GitHub workflow called `pull-translations`.

### Manual updates

If for some reason you wish to update the translation files by hand, several steps need to be done in the correct order to correctly merge and maintain them.

There are scripts available for these, so usually the process will be as follows:

Expand All @@ -373,6 +381,15 @@ So for example if your local language is New Zealand English (en_NZ), and you wa

You can also change the language in game options if both are installed.

### Language stats

We also store some statistics for how complete each translation is, in
`src/lang_stats.inc`. This can be used for example to show end users some
information about the translations.

These stats are also normally updated by the GitHub workflow, but can be
updated by hand via `lang/update_stats.sh`.

[1]: https://www.transifex.com/cataclysm-dda-translators/cataclysm-dda/
[2]: https://discourse.cataclysmdda.org/c/game-talk/translations-team-discussion
[3]: https://docs.transifex.com/
Expand Down
53 changes: 20 additions & 33 deletions lang/merge_po.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,32 @@ then
fi
fi

function merge_lang
{
f="lang/incoming/${n}.po"
o="lang/po/${n}.po"
if [ -f ${o} ]
then
echo "merging ${f}"
msgcat -F --use-first ${f} ${o} -o ${o} && rm ${f}
else
echo "importing ${f}"
mv ${f} ${o}
fi

# merge lang/po/cataclysm-dda.pot with .po file
echo "updating $o"
msgmerge --sort-by-file --no-fuzzy-matching $o lang/po/cataclysm-dda.pot | msgattrib --sort-by-file --no-obsolete -o $o
}

# merge incoming translations for each language specified on the commandline
if [ $# -gt 0 ]
then
for n in $@
do
if [ -f lang/incoming/${n}.po ]
then
if [ -f lang/po/${n}.po ]
then
echo "merging lang/incoming/${n}.po"
msgcat -F --use-first lang/incoming/${n}.po lang/po/${n}.po -o lang/po/${n}.po && rm lang/incoming/${n}.po
else
echo "importing lang/incoming/${n}.po"
mv lang/incoming/${n}.po lang/po/${n}.po
fi
merge_lang "${n}"
fi
done
# if nothing specified, merge all incoming translations
Expand All @@ -35,30 +46,6 @@ then
for f in lang/incoming/*.po
do
n=`basename ${f} .po`
if [ -f lang/po/${n}.po ]
then
echo "merging ${f}"
msgcat -F --use-first ${f} lang/po/${n}.po -o lang/po/${n}.po && rm ${f}
else
echo "importing ${f}"
mv ${f} lang/po/${n}.po
fi
done
fi

# merge lang/po/cataclysm-dda.pot with .po file for each specified language
if [ $# -gt 0 ]
then
for n in $@
do
echo "updating lang/po/${n}.po"
msgmerge --sort-by-file --no-fuzzy-matching lang/po/${n}.po lang/po/cataclysm-dda.pot | msgattrib --sort-by-file --no-obsolete -o lang/po/${n}.po
done
# otherwise merge lang/po/cataclysm-dda.pot with all .po files in lang/po
else
for f in lang/po/*.po
do
echo "updating $f"
msgmerge --sort-by-file --no-fuzzy-matching $f lang/po/cataclysm-dda.pot | msgattrib --sort-by-file --no-obsolete -o $f
merge_lang "${n}"
done
fi
2 changes: 1 addition & 1 deletion lang/update_pot.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ fi
echo "> Merging translation templates"
msgcat -o lang/po/cataclysm-dda.pot --use-first lang/po/json.pot lang/po/gui.pot
if [ ! -f lang/po/cataclysm-dda.pot ]; then
echo "Error in merging translatoin templates. Aborting."
echo "Error in merging translation templates. Aborting."
exit 1
fi

Expand Down
29 changes: 29 additions & 0 deletions lang/update_stats.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/usr/bin/env bash

if [ ! -d lang/po ]
then
if [ -d ../lang/po ]
then
cd ..
else
echo "Error: Could not find lang/po subdirectory."
exit 1
fi
fi

for f in lang/po/*.po
do
n=`basename ${f} .po`
o="lang/po/${n}.po"
echo "getting stats for ${n}"
num_translated=$( \
msgattrib --translated "${o}" 2>/dev/null | grep -c '^msgid')
num_untranslated=$( \
msgattrib --untranslated "${o}" 2>/dev/null | grep -c '^msgid')
mkdir -p lang/stats
printf '{"%s"sv, %d, %d},\n' \
"${n}" "$((num_translated-1))" "$((num_untranslated-1))" \
> lang/stats/${n}
done

cat lang/stats/* > src/lang_stats.inc
19 changes: 19 additions & 0 deletions src/lang_stats.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#include <initializer_list>
#include <lang_stats.h>

using namespace std::literals::string_view_literals;

static constexpr std::initializer_list<lang_stats> all_lang_stats = {
#include <lang_stats.inc>
};

const lang_stats *lang_stats_for( std::string_view lang )
{
for( const lang_stats &l : all_lang_stats ) {
if( l.name == lang ) {
return &l;
}
}

return nullptr;
}
18 changes: 18 additions & 0 deletions src/lang_stats.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#ifndef CATA_SRC_LANG_STATS_H
#define CATA_SRC_LANG_STATS_H

#include <string_view>

struct lang_stats {
std::string_view name;
int num_translated;
int num_untranslated;

float percent_translated() const {
return 100.0 * num_translated / ( num_translated + num_untranslated );
}
};

const lang_stats *lang_stats_for( std::string_view lang );

#endif // CATA_SRC_LANG_STATS_H
27 changes: 27 additions & 0 deletions src/lang_stats.inc
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{"ar"sv, 5, 85479},
{"cs"sv, 2558, 82926},
{"da"sv, 230, 85254},
{"de"sv, 30775, 54709},
{"el"sv, 336, 85148},
{"es_AR"sv, 70464, 15020},
{"es_ES"sv, 75721, 9763},
{"fil_PH"sv, 45, 85439},
{"fr"sv, 43094, 42390},
{"ga_IE"sv, 4, 85480},
{"hu"sv, 21727, 63757},
{"id"sv, 647, 84837},
{"is"sv, 292, 85192},
{"it_IT"sv, 17861, 67623},
{"ja"sv, 82685, 2799},
{"ko"sv, 45566, 39918},
{"nb"sv, 3817, 81667},
{"nl"sv, 1162, 84322},
{"pl"sv, 68898, 16586},
{"pt_BR"sv, 29182, 56302},
{"ro"sv, 419, 85065},
{"ru"sv, 74906, 10578},
{"sr"sv, 175, 85309},
{"tr"sv, 471, 85013},
{"uk_UA"sv, 6708, 78776},
{"zh_CN"sv, 81014, 4470},
{"zh_TW"sv, 52200, 33284},
80 changes: 48 additions & 32 deletions src/options.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
#include "generic_factory.h"
#include "input.h"
#include "json.h"
#include "lang_stats.h"
#include "line.h"
#include "mapsharing.h"
#include "output.h"
Expand Down Expand Up @@ -1302,36 +1303,49 @@ std::vector<options_manager::id_and_option> options_manager::get_lang_options()
{
std::vector<id_and_option> lang_options = {
{ "", to_translation( "System language" ) },
// Note: language names are in their own language and are *not* translated at all.
// Note: Somewhere in Github PR was better link to msdn.microsoft.com with language names.
// http://en.wikipedia.org/wiki/List_of_language_names
{ "en", no_translation( R"(English)" ) },
{ "ar", no_translation( R"(العربية)" )},
{ "cs", no_translation( R"(Český Jazyk)" )},
{ "da", no_translation( R"(Dansk)" )},
{ "de", no_translation( R"(Deutsch)" ) },
{ "el", no_translation( R"(Ελληνικά)" )},
{ "es_AR", no_translation( R"(Español (Argentina))" ) },
{ "es_ES", no_translation( R"(Español (España))" ) },
{ "fr", no_translation( R"(Français)" ) },
{ "hu", no_translation( R"(Magyar)" ) },
{ "id", no_translation( R"(Bahasa Indonesia)" )},
{ "is", no_translation( R"(Íslenska)" )},
{ "it_IT", no_translation( R"(Italiano)" )},
{ "ja", no_translation( R"(日本語)" ) },
{ "ko", no_translation( R"(한국어)" ) },
{ "nb", no_translation( R"(Norsk)" )},
{ "nl", no_translation( R"(Nederlands)" )},
{ "pl", no_translation( R"(Polski)" ) },
{ "pt_BR", no_translation( R"(Português (Brasil))" )},
{ "ru", no_translation( R"(Русский)" ) },
{ "sr", no_translation( R"(Српски)" )},
{ "tr", no_translation( R"(Türkçe)" )},
{ "uk_UA", no_translation( R"(український)" )},
{ "zh_CN", no_translation( R"(中文 (天朝))" ) },
{ "zh_TW", no_translation( R"(中文 (台灣))" ) },
};

constexpr std::array<std::pair<const char *, const char *>, 25> language_names = {{
// Note: language names are in their own language and are *not* translated at all.
// Note: Somewhere in Github PR was better link to msdn.microsoft.com with language names.
// http://en.wikipedia.org/wiki/List_of_language_names
{ "en", R"(English)" },
{ "ar", R"(العربية)" },
{ "cs", R"(Český Jazyk)" },
{ "da", R"(Dansk)" },
{ "de", R"(Deutsch)" },
{ "el", R"(Ελληνικά)" },
{ "es_AR", R"(Español (Argentina))" },
{ "es_ES", R"(Español (España))" },
{ "fr", R"(Français)" },
{ "hu", R"(Magyar)" },
{ "id", R"(Bahasa Indonesia)" },
{ "is", R"(Íslenska)" },
{ "it_IT", R"(Italiano)" },
{ "ja", R"(日本語)" },
{ "ko", R"(한국어)" },
{ "nb", R"(Norsk)" },
{ "nl", R"(Nederlands)" },
{ "pl", R"(Polski)" },
{ "pt_BR", R"(Português (Brasil))" },
{ "ru", R"(Русский)" },
{ "sr", R"(Српски)" },
{ "tr", R"(Türkçe)" },
{ "uk_UA", R"(український)" },
{ "zh_CN", R"(中文 (天朝))" },
{ "zh_TW", R"(中文 (台灣))" },
}
};

for( const auto& [lang_id, lang_name] : language_names ) {
std::string name = lang_name;
if( const lang_stats *stats = lang_stats_for( lang_id ) ) {
name += string_format( _( " <color_dark_gray>(%.1f%%)</color>" ),
stats->percent_translated() );
}
lang_options.emplace_back( lang_id, no_translation( name ) );
}

std::unordered_set<std::string> lang_list = get_langs_with_translation_files();

std::vector<id_and_option> options;
Expand Down Expand Up @@ -1679,7 +1693,9 @@ void options_manager::add_options_interface()
};

add( "USE_LANG", "interface", to_translation( "Language" ),
to_translation( "Switch language." ), options_manager::get_lang_options(), "" );
to_translation( "Switch language. Each percentage is the fraction of strings translated "
"for that language." ),
options_manager::get_lang_options(), "" );

add_empty_line();

Expand Down Expand Up @@ -3199,7 +3215,7 @@ std::string options_manager::show( bool ingame, const bool world_options_only, b

const int iWorldOffset = world_options_only ? 2 : 0;
int iMinScreenWidth = 0;
const int iTooltipHeight = 6;
const int iTooltipHeight = 7;
int iContentHeight = 0;
bool recalc_startpos = false;

Expand Down Expand Up @@ -3383,8 +3399,8 @@ std::string options_manager::show( bool ingame, const bool world_options_only, b
const std::string name = utf8_truncate( name_value.first.s, name_width );
mvwprintz( w_options, point( name_col + 3, line_pos ), name_value.first.col, name );

const std::string value = utf8_truncate( name_value.second.s, value_width );
mvwprintz( w_options, point( value_col, line_pos ), name_value.second.col, value );
trim_and_print( w_options, point( value_col, line_pos ), value_width,
name_value.second.col, name_value.second.s );

opt_line_map.emplace( i, inclusive_rectangle<point>( point( name_col, line_pos ),
point( value_col + value_width - 1, line_pos ) ) );
Expand Down

0 comments on commit e129921

Please sign in to comment.