Accentueted characters on linux

[WildCerebrus]

Hi, I'm on manuskript 0.5.0 on Ubuntu 16.04

whenever I try to write an "ê"(resp. an "â", "ô" or "î") manuskript replace it by "e" (resp. "a","o","i") it's really inconvenient in french

by the way I can copy-paste those characters even if I can't write it directly

[gedakc]

Are you running manuskript from the binary package?

If so then this is a known issue with our PyInstaller packages, as noted in the wiki under the Install Binary section.

A few work-arounds are listed (you discovered one). The fix for now is to run manuskript from source code.

[olivierkes]

Hi @WildCerebrus, thanks for reporting. Unfortunately we haven't been able yet to find the cause of that bug. Et c'est pénible !

I played for a while, trying to remove as much libs as possible from package, and was unsuccessful to find the bug, but I narrowed the files down. Here is a list of all files that can be removed in the package.

For now, the easiest would be to run from source as @gedakc noted. It's not that difficult.

[olivierkes]

Observation: I made a simple PyQt test file with only a QPlainTextEdit (the base class I use for the text area). I packaged it with PyInstaller, and it runs fine, and accepts accentuated characters. I tried to tweak the spec file, but wasn't able to reproduce.

Conclusion: it's most probably a bug in my code, and not in PyInstaller.

BTW, can we reproduce in windows/osx packages?

Test file:

#!/usr/bin/python3
# -*- coding: utf-8 -*-

import sys
from PyQt5.QtWidgets import QApplication, QPlainTextEdit

if __name__ == '__main__':

    app = QApplication(sys.argv)

    w = QPlainTextEdit()
    w.resize(250, 150)
    w.move(300, 300)
    w.show()

    sys.exit(app.exec_())

[olivierkes]

Observation 2: The bug is present in every text area, even in "Open file" dialog, or native simple QLineEdit widgets that I haven't subclassed and modified.

Conclusion 2: It's most probably a bug in packaging, not in my code.

(So on average, it's most probably a bug. But we don't know where.)

[WildCerebrus]

I think you can't reproduce on windows (I use it finely) but I don't know about osx Le 11 nov. 2017 23:26, "Olivier" <notifications@github.com> a écrit :

…

*Observation 2:* The bug is present in every text area, even in "Open file" dialog, or native simple QLineEdit widgets that I haven't subclassed and modified. *Conclusion 2:* It's most probably a bug in packaging, not in my code. (So on average, it's most probably a bug. But we don't know where.) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#207 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJlhZJqt1s_Eo4YX67syblwmgXZFjNb3ks5s1h8dgaJpZM4QaelJ> .

[olivierkes]

I think you can't reproduce on windows (I use it finely) but I don't know about osx

Ok thanks !

New test

I got an idea this morning: package older versions.

Methodology

• Download the source code of each release
• Extract
• Run make run to be sure it works
• Run pyinstaller bin/manuskript if there is no .spec file, otherwise pyinstaller manuskript.spec (more exactly, I run python3 /usr/local/bin/pyinstalller manuskript.spec)
• If it fails, do some tweakings (see details below)
• Once built, run ./dist/manuskript/manuskript, and test if I can enter accentuated characters in the welcome widget.

Results

• 0.1.0: accentuated characters works.
  Note: problem with packaging the libodt, so had to remove by commenting 5 lines in manuskript/exporter/__init__.py

• 0.2.0: accentuated characters works.
  Note: problem with packaging the libodt, so had to remove by commenting 5 lines in manuskript/exporter/__init__.py

• 0.3.0: accentuated characters works.
  Note: problem with packaging the libodt, so had to remove by commenting 5 lines in manuskript/exporter/__init__.py

• 0.4.0: accentuated characters works (not only in welcome widget but also in texts)

• 0.5.0: accentuated characters works. (not only in welcome widget but also in texts)

Analysis and hypotheses

Not at all the expected outcome, but interresting nonetheless !

I was expecting that at some release it would stop working, but it does not. So I see three options:

1. There is something wrong in my testing methodology
2. The issue is on the computer creating the packages (like missing something), but what would it be?
3. It works because it is run on the same computer as the one who created it.

Unexpected discovery

In order to test 3., I zipped the folder to post it here for testing. But I tested it before posting, and lo and behold! it doesn't work!

• I zipped the file using Dolphin's right-click > Compress > Here as zip (Dolphin being KDE's default file browser).
• I tested by: deleting the dist dir, moving the archive to a new directory, extracting, running.

I'm calling that progress!!

I'll do further testing later on. (Sorry for the long post, most of it is unnecessary now, but I keep it for myself for the log)

[gedakc]

Wow, you've sure done lots of testing @olivierkes.

As best I can determine, the problem with not being able to enter accented characters directly from a keyboard arises with the PyInstaller binary package only.

One of the challenges with trying to determine which PyInstaller package file can be removed is that it also seems to depend upon which packages are installed in the GNU/Linux distro you are testing. In my investigations I've found that a file required in one distro might not be required in another.

Unfortunately I have not been able to discover any other clues when searching the Internet to see if other projects have encountered the same issue with accented characters.

I am also unable to recreate the accent test because I only have US Keyboards. :(

2. The issue is on the computer creating the packages (like missing something), but what would it be?

Reading your point 2 made me think about checking for base language support in the 64-bit Debian 8 Jessie VM I use to build the PyInstaller packages.

I came across the following link on Debian and keyboards:

Debian - Keyboard

In this article the example /etc/default/keyboard file indicates multiple languages with the line:

XKBLAYOUT="us,de,fr,ua,ru"

On my VM I only have one language listed:

XKBLAYOUT="us"

This led me to try adding additional keyboard layouts, and then create the PyInstaller package.

Add a foreign keyboard layout to Gnome in Debian 8

I ran the following command:

sudo apt-get install keyboard-configuration

And my distro said it was already installed.

I also looked at the following man page for information about XKBLAYOUT:

keyboard(5)

I edited my line to be the same as the example line above:

XKBLAYOUT="us,de,fr,ua,ru"

Then I rebooted (in case it only takes effect after a reboot).

Next I created a PyInstaller package and then compared the contents with the previous manuskript-0.5.0-linux64.zip package using meld.

Two files were listed as being different: base_library.zip and the manuskript executable.

I don't know if these differences are material, or if I need to run any additional commands.

Request for Help

Would someone be able to download and test manuskript-0.5.0-linux64-accent-test.zip with a non-US keyboard to see if accented characters can be directly entered?

[olivierkes]

Thanks for those tests ! I tried your package, and it still doesn't work. :( By the way, I also have only XKBLAYOUT="us".

I am also unable to recreate the accent test because I only have US Keyboards. :(

You can try using the compose key. In my test, it also fails. Those accentuated characters we're talking about are "two-steps" chars, for example é works because it's a single touch. But É doesn't work, because I have to type ´ then E. Do you have also "two-steps" characters on US keyboards? I think it's the same effect than the compose key, so in my tests I also try compose+--- and it should output —.

I tried to reproduce my tests of this morning, and wasn't able to get a faulty package..!?!
So here is one that works on my computer. Can somebody else try it?

[gedakc]

When running from source code, I can use the compose key (right windows key in my case) to generate accented characters.

• Press and hold compose, type forward quote ', type e, release compose to get è.
• Press and hold compose, type forward quote ', also press and hold shift, type e, release shift and release compose to get É.

The compose+--- combination did not work for me and instead simply output ---.

These key combinations do not work with my manuskript-0.5.0-linux64.zip file.
These key combinations do not work with my manuskript-0.5.0-linux64-accent-test.zip file.

These key combinations do work with your manuskript-0.5.0-linux64-1.zip file!

I wonder what is different with how you created your PyInstaller package?

[gedakc]

Whoa, now I'm able to get the compose+--- combination to work with your manuskript-0.5.0-linux64-1.zip file.

• Press and hold compose, type three hyphens -, release compose to get —.

Why didn't this work the first time I tried it???

[olivierkes]

Good, so now we have a way for you to test that bug. And we also have a package that works, though we don't know why.

I really can't reproduce the faulty package I got this morning... I'm trying to think of everything that I might have changed:

• I thought it might be the python version (I'm using pyenv), but I tried with both python version I have installed (3.5.2 and 3.6.0), and no differences
• I have upgraded pip to the latest version, could it make a difference?
• I have tried to upgrade pyinstaller, but I think it as already up to date (3.3)

I see two main differences between your manuskript-0.5.0-linux64.zip and my manuskript-0.5.0-linux64-1.zip:

1. The python version: 3.4 in yours, 3.5 in mine.
2. The way qt/pyqt is bundled: in a qt5_plugins folder in yours, in a PyQt5/Qt/plugins folder in mine.
3. Edit: the Qt/PyQt version: 5.3.2 vs 5.5.1

I'll try to see what causes the difference in (2.). Can you try maybe to test packaging with a different python version? (pyenv is version convenient for that)

[gedakc]

I tried with Python 3.5.4 and PyQt 5.8.2, but no luck. I am still unable to enter accented characters in the package I built. :(

Which exact versions of Python did you use?

For reference the steps I used to install pyenv on my 64-bit Debian 8 GNOME VM were as follows:

# pyenv and PyInstaller
#
# https://github.com/pyinstaller/pyinstaller/wiki/Development

# clone pyenv repository
git clone https://github.com/yyuu/pyenv.git ~/.pyenv

# git clone virtualenv plugin
git clone https://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

# add to .bashrc or .zshrc or .bash_pyenv:
#----- 8< -----
# Add 'pyenv' to PATH.
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"

# Enable shims and autocompletion for pyenv.
eval "$(pyenv init -)"
# Load pyenv-virtualenv automatically by adding
# # the following to ~/.zshrc:
#
eval "$(pyenv virtualenv-init -)"
#----- >8 -----

# Add pyenv environment vars
. ~/.bash_pyenv

#
#    Install python version with shared libpython (necessary for PyInstaller to work)
#
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.5.4

# virtualenv
pyenv virtualenv 3.5.4 pyenv354

# activate virtualenv
pyenv activate pyenv354

# deactivate virtualenv
#pyenv deactivate


#
# Install manuskript python modules
#
pip3 install pyqt5==5.8.2
pip3 install lxml markdown pyenchant


#
# Test manuskript works
#
bin/manuskript


#
# Install pyinstaller
#
pip3 install pyinstaller

#
# Build pyinstaller package
#
python3 /home/user/.pyenv/versions/3.5.4/envs/pyenv354/bin/pyinstaller manuskript.spec

[olivierkes]

This could quickly get very frustrating :)

Which exact versions of Python did you use?

3.5.2 and 3.6.0

What's your version of pyinstaller? (mine is 3.3)

Can you see in your packages if it uses the qt5_plugins folder structure, or PyQt5/Qt/plugins?