ansible: environment difference while running debops playbooks

[dw]

Found while working on #110. My bashrc environment sets en_GB.UTF-8. When running with Mitogen, the run completes, but with non-Mitogen, then run fails with:

TASK [debops.sshd : Check if OpenSSH is installed (or half-installed)] *************************************************************************************************************
fatal: [35.200.184.46]: FAILED! => {"changed": false, "failed": true, "failed_when_result": true, "rc": 0, "stderr": "Shared connection to 35.200.184.46 closed.\r\n", "stdout": "bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)\r\ninstalled\r\n", "stdout_lines": ["bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)", "installed"]}

Because the Ansible master's environment is getting copied across somehow, and the locale data was missing on the target box. Figure out what's happening

[stefangweichinger]

had that debops.sshd issue as well here. Just fyi.

[drybjed]

If it works with Mitogen (I wonder how) but doesn't work without it, it seems like the dpkg command doesn't like to use non-existent locales. Your en_gb.UTF-8 locale is passed through SSH because by default LC_* environment variables are whitelisted by sshd.

Perhaps setting the environment as LC_ALL: C for this task could solve it.

[dw]

I just pushed issue_122.yml which I was using earlier to try and reproduce the difference. Running vimdiff on the output with/without Mitogen shows no environment difference. Probably I'm still doing something wrong to reproduce the original scenario, will look at it later in the week.

No matter what the current behaviour is, Mitogen needs to emulate it. It needs to let both the SSH invocation and the sudo invocation do whatever horridness is done to the environment and reproduce exactly that, otherwise more playbooks will be broken behind closed doors

[drybjed]

@dw Ah, so you want to make sure that Mitogen fails the same way that Ansible fails with the same environment, good idea.

[dw]

Discoveries so far:

• OS X ssh_config does not have SendEnv by default
• Debian ssh_config has SendEnv by default in /etc/ssh/ssh_config
• Fedora ssh_config sets SendEnv in /etc/ssh/ssh_config.d/05-redhat.conf

As usual the correct behaviour isn't exactly clear. I dislike having bits of the environment leak into the tooling (it makes it impossible to test), but we have to emulate Ansible.

Relevant bugs:

• LC_MESSAGES should also be forced: Ansible may not behave the same depending on your ssh_config SendEnv values ansible/ansible#9635
• Ansible permits ssh to send local LANG and LC_ALL env vars (file misencoding, filename corruption) ansible/ansible#10698

Relevant spaghetti:

• C.CONTROLER_LANG is set from LANG environment variable
• C.DEFAULT_MODULE_LANG defaults to C.CONTROLER_LANG unless ANSIBLE_MODULE_LANG is present in the environment, or module_lang is present in [defaults] in ansible.cfg
• plugins/shell/__init__.py uses C.DEFAULT_MODULE_LANG or (lol) the LANG environment variable if C.DEFAULT_MODULE_LANG is unset, to build a dict of defaults environment variables for any shell command. The dict contains LANG, LC_ALL and LC_MESSAGES forced to that value, but it only does this if C.DEFAULT_MODULE_SET_LOCALE is set, which is set if the ANSIBLE_MODULE_SET_LOCALE environment variable is set, or the module_set_locale key exists in [defaults] of ansible.cfg.
• plugins/action/__init__.py uses ShellBase.env_prefix() from _compute_environment_string() to build the snippet for setting the environment using the active shell, but _compute_environment_string() does not return those variables in its output dictionary, only in the returned snippet.
• plugins/action/__init__.py calls _compute_environment_string() from _execute_module() only. _low_level_execute_command() does not get this environment treatment

Sphaghetti round 2:

• actions/script.py (which is the failing action) is not actually using _execute_module(), but calling _compute_environment_string() directly.

Can't reproduce this locally, going to have to setup two Gcloud boxes again

[dw]

@stefangweichinger do you still have access to your setup that triggered the locale error? If so, can you please --start-at="Check if OpenSSH is installed (or half-installed)" and attach "-vvv" output both for Ansible and for Mitogen.

If not don't worry, I'll pick this up next week

[stefangweichinger]

@dw yes, I have. pls more details: with or without your new debug variable exported, mitogen.log from controller?

[dw]

Just -vvv is fine, MITOGEN_ROUTER_DEBUG is only useful for lower level IO issues

[stefangweichinger]

ansible-playbook ~/.local/share/debops/debops/ansible/playbooks/common.yml -vvv --start-at="Check if OpenSSH is installed (or half-installed)" -l tx100 > debops_common.log

debops_common.log

can't find a mitogen.log in /tmp now, though

note: afk soon ...