Feature: allow Ansible provisioner to specify usage of WinRM instead of SSH #3911

Holmistr · 2016-09-22T08:35:49Z

My usecase is to create Windows image using Ansible. The problem is that I need to connect to Windows image via WinRM. Ansible already has this capability, you only specify "ansible_connection: winrm" and other parameters in the inventory file.

However, Packer's Ansible provisioner doesn't allow to add this information to the inventory file. Packer create temporary inventory file which it uses further. As stated in the documentation: "It dynamically creates an Ansible inventory file configured to use SSH ... ". Also, when looking at the code of Ansible provisioner (https://github.com/mitchellh/packer/blob/master/provisioner/ansible/provisioner.go#L252), I see that there is no option to either specify ansible_connection properties, or provide custom inventory file.

This feature could be quite useful and I would say not very hard to implement. If you can confirm that this makes sense to implement and we agree on the way how to do it (either provide capability to specify custom inventory file or just add new properties like "ansible_connection", etc.), I would be happy to implement it (or at least try).

rickard-von-essen · 2016-09-22T08:39:49Z

@bhcleek do you know if this is already supported, maybe on master?

Holmistr · 2016-09-22T08:43:26Z

Hi @rickard-von-essen , thanks for your comment. As I linked the code, I think it's quite clear that this is not implemented, however maybe I'm overlooking something, so let's wait for the comment. Thanks for such a quick respose :)

rickard-von-essen · 2016-09-22T09:14:26Z

This is a bit complicated since ansible communicates with a ssh proxy that in turn forwards the connection to the VM. But it should now work on master for docker which also uses a special communicator, if I understand correctly.

bhcleek · 2016-09-22T15:06:44Z

Yes, this should work on master now.

rickard-von-essen · 2016-09-22T15:11:10Z

👍

Holmistr · 2016-09-23T14:35:28Z

Thanks for you responses @bhcleek . In that case, could you please provided any documentation/example, how to use this feature? Since it doesn't work for me even when I try to build fresh master, but I haven't configured anything.

My Packer JSON file looks like this: http://pastebin.com/qh9R00Qr
Corresponding Ansible file just does simple creation of a file via PowerShell. When I execute the Ansible manually (via ansible-playbook command) on a running instance, everything works fine. To make this working, I had to create Ansible group_vars/windows.yaml file with ansible_connection: winrm property and everything started working (manual execution of ansible). However, I don't understand how Packer would pick up that it should use WinRM, could you show me the code that should do this for Ansible?

Output of my image build: http://pastebin.com/sC8wRSrs

So question - if this feature is already implement, how to use it?

bhcleek · 2016-09-25T21:13:19Z

@Holmistr Packer communicates with the node, because the builder's communicator is set to to winrm. Ansible will not be communicating directly with the node; instead, it will use SSH to connect to the provisioner. The provisioner takes care of communicating with the node.

Your packer template looks fine to me; I'm working on some tests to verify that the provisioner works correctly in this configuration. In the meantime, have you tried using Ansible's raw module? Adding "ansible_extra_vars": ["-vvvv"] to your template and running packer with PACKER_LOG=1 will also help us analyze the provisioning of your windows node.

Holmistr · 2016-09-26T07:43:02Z

@bhcleek Yes, I tried the raw module.
After enabling more logging as you suggested, I got just a little more info:

openstack: <127.0.0.1> ESTABLISH WINRM CONNECTION FOR USER: jholusa on PORT 44902 TO 127.0.0.1
openstack: <127.0.0.1> ESTABLISH WINRM CONNECTION FOR USER: jholusa on PORT 44902 TO 127.0.0.1
2016/09/26 09:27:48 ui: openstack: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "ssl: auth method ssl requires a password", "unreachable": true}

I'm not very familiar with the process, but it looks like it's trying to establish WinRM connection to the local proxy, is that correct? I'm sorry, I'm still a bit confused here, hence I cannot diagnose it properly.

chris-smith-zocdoc · 2016-09-29T02:28:40Z

I've setup a simple test repo https://github.com/chris-smith-zocdoc/packer-windows-example for this issue. If I'm doing something wrong/dumb please let me know, the rest of this is based on the assumption that what I'm doing in the example repo is correct.

AFAIK its unlikely that the current implementation will work unless there is a way to have Ansible send windows commands over the ssh connection.

Here is my packer log

The interesting lines

2016/09/29 01:23:17 packerio: 2016/09/29 01:23:17 new exec request: /bin/sh -c '( umask 22 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1475112197.48-12092160621548 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1475112197.48-12092160621548 `" )'
2016/09/29 01:23:17 packerio: 2016/09/29 01:23:17 [INFO] starting remote command: /bin/sh -c '( umask 22 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1475112197.48-12092160621548 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1475112197.48-12092160621548 `" )'
2016/09/29 01:23:17 [INFO] 0 bytes written for 'stdout'
2016/09/29 01:23:17 [INFO] 44 bytes written for 'stderr'
2016/09/29 01:23:17 [INFO] RPC client: Communicator ended with: 1
2016/09/29 01:23:17 [INFO] RPC endpoint: Communicator ended with: 1
2016/09/29 01:23:17 [INFO] 0 bytes written for 'stdin'

Notice that its trying to issue a command to the windows host using /bin/sh, this obviously won't work on a windows machine.

If you take a look at the Ansible winrm connector, it does a bunch of things when executing winrm commands which doesn't happen over the ssh connection.

It looks like it might be possible to force the ssh connector to use the powershell shell type but I'm not familiar enough with the Ansible codebase to make that determination. I also wouldn't be surprised if that just brought on a new slew of bugs in Ansible because I imaging anyone who is doing windows management with Ansible is using winrm.

Running Ansible from outside of packer on the same host, you'll notice that the commands change to be more powershelly

I'm not familiar with either of these codebases, but from what I can tell we have the following options:

Force Ansible to send powershell commands over the ssh connection
- This could expose a bunch of bugs on the Ansible side because most people doing Windows management are probably using winrm
Implement a Winrm proxy to forward from Ansible -> proxy -> target
- Maybe there is a go implemenation already?
Expose the ip address from the packer connector to the Ansible provisioner, allowing Ansible to talk directly to the target machine
- Mentioned previously in Provide host based Ansible provisioner #1555

bhcleek · 2016-09-29T03:12:56Z

@chris-smith-zocdoc My testing shows the same results that you're seeing, but I think there may be a way forward. ansible/ansible#14274, ansible/ansible#13490, and http://docs.ansible.com/ansible/intro_configuration.html#executable hint that it may be possible to force ansible to use the shell of our choice; I'll keep looking into it.

Regarding the other two alternatives that you provided, implementing a WinRM proxy may be possible. The last alternative, exposing the IP address from the communicator interface is unlikely to happen, because not all the builders communicate over TCP/IP.

bhcleek · 2016-09-29T03:38:45Z

https://github.com/ansible/ansible/blob/stable-2.1/lib/ansible/plugins/connection/__init__.py#L173-L221 and https://github.com/ansible/ansible/blob/600915aa97cd70dc3b6c5e5a3d1944f48c14597e/docsite/rst/intro_inventory.rst#list-of-behavioral-inventory-parameters give us details about the values that may need to be configured using extra_arguments and/or ansible_env_vars. I suspect that ansible_shell_type should be powershell and that ansible_shell_executable should also be set to something, but I'm not sure what. Maybe powershell?

chris-smith-zocdoc · 2016-09-29T04:32:05Z

@bhcleek

I was able to force the executable to be powershell and Ansible made it a little further. This time it failed to run scp (I assume Ansible is copying files to the target machine) See the updated log

I'm currently running an older version of Ansible 2.0.2.0, I'll update to 2.1 and try some combination of those parameters

not all the builders communicate over TCP/IP.

For my own knowledge, what packer builders don't support TCP/IP?

bhcleek · 2016-09-29T15:59:49Z

That's great news! The scp call you see isn't from Ansible to the provisioner; it's from Packer to the node; you can safely ignore it.

Can you try setting the ansible_shell_type to powershell, too?

For my own knowledge, what packer builders don't support TCP/IP?

The docker builder, for instance, doesn't use SSH to communicate with Docker. Currently it runs docker commands, but there's a pending change (maybe already merged?) that will use the Docker API instead. Also, any number of custom builders may use their own communicator.

chris-smith-zocdoc · 2016-09-29T18:14:09Z

I tried setting the shell type to powershell but this caused an exception inside ansible

The docker builder, for instance, doesn't use SSH to communicate with Docker.

Ansible also built a docker connector so I don't think that's an issue. I see a lot of duplicate work between the two projects which seems unfortunate.

chris-smith-zocdoc · 2016-10-04T05:27:18Z

I took a look at the WinRM protocol, turns out its all done over HTTP which is really convenient for us. We should be able to get this to work by setting up a HTTP proxy between Ansible and the target machine. Example of someone doing this in nginx

There are plenty of http proxy packages available, including the one in the std lib https://golang.org/pkg/net/http/httputil/#ReverseProxy and this one built on top of it https://github.com/elazarl/goproxy

rickard-von-essen · 2016-10-04T07:32:46Z

It would be awesome to implement a WinRM proxy that can forward WinRM, it should be fairly easy. We could extract the ssh proxy and the WinRM proxy into a utility and allow things like shell-local to get access to connection details towards a proxy. By using a proxy we ensure that stuff like ssh_bastion_host config still works without handing over all this complexity to the scripts in shell-local.

bhcleek · 2016-10-04T16:38:44Z

Keep in mind that the ansible-remote provisioner is an adapter, not a proxy.

@rickard-von-essen are you suggesting that a communicator bypass be put in place?

I'd like to take a little more time to see if we can find the right set of variables that would allow Ansible to be configured so that Ansible could connect to the provisioner using SSH, but using Powershell on the remote node.

rickard-von-essen · 2016-10-04T17:00:21Z

@bhcleek Sure if you ask a Software Engineer it's an adapter, but ask a Network Engineer and he'll call it a proxy ;-)

are you suggesting that a communicator bypass be put in place?

No I was just thinking aloud that your adapter/proxy could be use with shell-local to allow something like

{
  "type": "shell-local",
  "inline": [ "git push ssh://git@localhost:$SSH_PORT" ]
}

Where $SSH_PORT is a port where the adapter receives connections and forwards it to a Communicator.

But this is not the solution for this issue right now.

bhcleek · 2016-10-04T17:18:53Z

Thanks for the clarification. Is there another forum where we could talk about it?

bhcleek · 2016-10-07T15:18:59Z

@chris-smith-zocdoc your latest attempt made some good progress, and gives me some confidence that we're getting close to a solution. I think we still don't have the ansible_shell_executable type set to the correct value. Have you tried setting it to nothing (e.g. empty)?

I'll spend some time trying to provision a Windows machine with Ansible this weekend.

chris-smith-zocdoc · 2016-10-07T15:27:09Z

@bhcleek I'll try that and report back. We can probably also reach out to the Ansible team to see if they can point is in the right direction

bhcleek · 2016-10-10T04:51:57Z

I'm happy to report that I've had success provisioning a Windows machine using the ansible provisioner by adding "--extra-vars", "ansible_shell_type=powershell ansible_shell_executable=" to the --extra-arguments property:

    "provisioners": [
      {
        "type":  "ansible",
        "playbook_file": "./win-playbook.yml",
        "extra_arguments": [
          "-vvvv",
          "--extra-vars", "ansible_shell_type=powershell ansible_shell_executable="
        ]
      }
    ]

rickard-von-essen · 2016-10-10T06:16:31Z

@bhcleek that is great news. The we can just check if Comm.Type == "winrm" and then automatically add those extra vars. And dito for Docker. And make it clear in the docs that this is supported.

bhcleek · 2016-10-10T17:56:42Z

@rickard-von-essen the Communicator doesn't expose its type yet. Are you suggesting that we should add that?

rickard-von-essen · 2016-10-10T18:25:16Z

@bhcleek sorry have been programming to much on the builders where you have
access to config.Comm.Type.

On Oct 10, 2016 19:56, "Billie Cleek" notifications@github.com wrote:

@rickard-von-essen https://github.com/rickard-von-essen the Communicator
doesn't expose its type yet. Are you suggesting that we should add that?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#3911 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiCg6ijMxHCNM_MaQ16RYSnm1RSD4i8ks5qynxcgaJpZM4KDpZB
.

chris-smith-zocdoc · 2016-10-10T19:11:13Z

@bhcleek Do you have an example I can look at? Setting those arguments still gives me the error I was seeing previously. chris-smith-zocdoc/packer-windows-example@5746322

Which version of Ansible do you have installed?

ansible --version
ansible 2.1.2.0

The stack I'm seeing with those arguments

TASK [setup] *******************************************************************
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 124, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 460, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 601, in _execute_module
    tmp = self._make_tmp_path(remote_user)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 218, in _make_tmp_path
    result = self._low_level_execute_command(cmd, sudoable=False)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 732, in _low_level_execute_command
    cmd = self._connection._shell.append_command(cmd, 'sleep 0')
AttributeError: 'ShellModule' object has no attribute 'append_command'
fatal: [default]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}

bhcleek · 2016-10-11T18:09:45Z

I'm using the same version of ansible, 2.1.2.0. Your playbook looks like it should work. However, I am seeing the same error as you under some conditions. I'm writing some tests to verify the WinRM functionality and will let you know when I know more.

bhcleek · 2016-10-14T03:52:48Z

After troubleshooting some more, I've realized that so far I am only able to get the raw module to work. @chris-smith-zocdoc I think you'll see your playbook work if you'll turn off fact gathering (e.g. gather_facts: no).

bhcleek · 2016-10-30T19:25:50Z

For everyone following this, I think the problem here may actually lie in Ansible. I'm working to resolve it, and will post an update or submit a PR when I have a solution.

Holmistr · 2016-11-21T11:10:55Z

Hi @bhcleek , I see that you've made some progress, nice! Do you have any update on this? I'm getting the same errors as mentioned above.

bhcleek · 2016-11-23T01:08:48Z

I have made progress. I'm dealing with one remaining issue, and then I need to refine the solution to be less hacky. So far I've had to make a very small change to the provisioner code and supply an Ansible connection type that's forked from the ssh connection type and edited with some small changes.

Unfortunately, Ansible's powershell shell type and ssh connection type don't play well together yet; the powershell shell type expects to be used only for winrm connections.

I'm going to keep at it, though, and will try to publish some code this weekend to show my progress.

Holmistr · 2016-11-23T09:28:26Z

Awesome news @bhcleek , I really appreciate this! :)

Assume the scp target is a file instead of a directory. Assuming the scp target is a file instead of a directory allows uploading files to a node being provisioned with the ssh communciator using sftp and with the winrm communicator. It is fully compatible with ansible; ansible communicators only allow for files to be uploaded (when the copy module is used to upload a directory, ansible walks the directory and uploads files one at a time). Update docuemntation to explain how to provision a Windows image. Extend tests that use ssh to communicate with the node to include single files, recursive copies, and content-only recursive copies. Add test to verify support for the winrm communicator. Remove the err argument from adapter.scpExec, because it was unused. Fixes hashicorp#3911

bhcleek · 2016-11-26T21:45:36Z

I've submitted a PR, #4209, that should resolve the incompatibility of using the ansible provisioner with nodes being provisioned using the winrm communicator. You can see how to hook this up in three files:

It's also documented in https://github.com/mitchellh/packer/blob/9201b2db606854c8c5fac3915ee855b4f7c5e3b5/website/source/docs/provisioners/ansible.html.md

Note that code changes were required, so you'll need to use a build that incorporates the code changes in #4209.

Assume the scp target is a file instead of a directory. Assuming the scp target is a file instead of a directory allows uploading files to a node being provisioned with the ssh communciator using sftp and with the winrm communicator. It is fully compatible with ansible; ansible communicators only allow for files (never directories) to be uploaded (when the copy module is used to upload a directory, ansible walks the directory and uploads files one at a time). Update documentation to explain how to provision a Windows image. Extend tests that use ssh to communicate with the node to include single files, recursive copies, and content-only recursive copies. Add test to verify support for the winrm communicator. Remove the err argument from adapter.scpExec, because it was unused. Fixes hashicorp#3911

transamfire85 · 2016-12-01T19:21:50Z

@bhcleek This worked wonderfully. Thank you so much!

dragan · 2016-12-05T06:05:16Z

@bhcleek Thanks for contributing a fix for using winrm within ansible and packer. I've been using a build based off your PR for a project and everything is working very well, except when using the win_reboot module.

I receive the following error:

vmware-iso: TASK [common : Rebooting] ******************************************************
vmware-iso: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'Connection' object has no attribute '_winrm_host'
vmware-iso: fatal: [default]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}

When running with -vvv, this is what I get:

vmware-iso: TASK [common : Rebooting] ******************************************************
vmware-iso: task path: main.yml:15
vmware-iso: An exception occurred during task execution. The full traceback is:
vmware-iso: Traceback (most recent call last):
vmware-iso:   File "/usr/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 119, in run
vmware-iso:     res = self._execute()
vmware-iso:   File "/usr/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 490, in _execute
vmware-iso:     result = self._handler.run(task_vars=variables)
vmware-iso:   File "/usr/local/lib/python2.7/site-packages/ansible/plugins/action/win_reboot.py", line 79, in run
vmware-iso:     winrm_host = self._connection._winrm_host
vmware-iso: AttributeError: 'Connection' object has no attribute '_winrm_host'
vmware-iso: fatal: [default]: FAILED! => {
vmware-iso:     "failed": true,
vmware-iso:     "msg": "Unexpected failure during module execution.",
vmware-iso:     "stdout": ""
vmware-iso: }

dragan · 2016-12-05T21:02:07Z

Update, I added the _winrm_host and _winrm_port attributes to the custom connection plugin for packer. However, the values will need to be set based on the VM, not the proxy/adapter.

I hard coded the values as a test. Is that information available to the connection plugin, so they can be set dynamically?

Everything works, except the ansible logging while it waits during reboot.

bhcleek · 2016-12-06T21:48:21Z

From the perspective of Ansible, the adapter is the thing being provisioned, so the _winrm_host and _winrm_port values would have to reflect the adapter values, not the VM.

However, rebooting using the win_reboot module will cause packer to lose the connection; if you need to reboot in the middle of provisioning, then use packer's windows restart provisioner instead of Ansible's win_reboot module.

dragan · 2016-12-07T00:49:02Z

Thanks @bhcleek.

I was hoping not to break up my playbook just for reboots when I'm using Ansible to do the provisioning. I do not know the history, but what was the use case that led to the proxy/adapter?

bhcleek · 2016-12-07T01:10:06Z

@dragan it's due to the (necessary) architecture of packer. RE: #3846 (comment)

rickard-von-essen added enhancement question provisioner/ansible-remote labels Sep 22, 2016

chris-smith-zocdoc mentioned this issue Sep 29, 2016

Ansible-Remote Provisioner and WinRM #3307

Closed

chris-smith-zocdoc mentioned this issue Oct 4, 2016

Expose connection settings in communicator to provisioners #3889

Closed

rickard-von-essen mentioned this issue Oct 6, 2016

RFC: Expose generic SSH / WinRM proxies for provisioner #3969

Closed

bhcleek mentioned this issue Nov 26, 2016

provisioner/ansible: assume scp target is file (resolves winrm incompatibility) #4209

Merged

mwhooker closed this as completed in #4209 Dec 9, 2016

fbsdmon mentioned this issue Feb 27, 2017

Ansible provisioner doesn't allow execution of raw powershell commands #4596

Closed

bhcleek mentioned this issue Dec 7, 2017

"Packer" Ansible connection for WinRM breaks on Ansible 2.3.0 #4904

Closed

rickard-von-essen mentioned this issue Mar 1, 2018

Allow user to access the randomly-generated WinRM password in the Powershell provisioner. #5895

Closed

ghost locked and limited conversation to collaborators Apr 5, 2020

Feature: allow Ansible provisioner to specify usage of WinRM instead of SSH #3911

Feature: allow Ansible provisioner to specify usage of WinRM instead of SSH #3911

Comments

Holmistr commented Sep 22, 2016

rickard-von-essen commented Sep 22, 2016

Holmistr commented Sep 22, 2016

rickard-von-essen commented Sep 22, 2016 • edited Loading

bhcleek commented Sep 22, 2016

rickard-von-essen commented Sep 22, 2016

Holmistr commented Sep 23, 2016

bhcleek commented Sep 25, 2016 • edited Loading

Holmistr commented Sep 26, 2016

chris-smith-zocdoc commented Sep 29, 2016

bhcleek commented Sep 29, 2016

bhcleek commented Sep 29, 2016 • edited Loading

chris-smith-zocdoc commented Sep 29, 2016

bhcleek commented Sep 29, 2016

chris-smith-zocdoc commented Sep 29, 2016

chris-smith-zocdoc commented Oct 4, 2016

rickard-von-essen commented Oct 4, 2016

bhcleek commented Oct 4, 2016

rickard-von-essen commented Oct 4, 2016 • edited Loading

bhcleek commented Oct 4, 2016

bhcleek commented Oct 7, 2016

chris-smith-zocdoc commented Oct 7, 2016

bhcleek commented Oct 10, 2016

rickard-von-essen commented Oct 10, 2016

bhcleek commented Oct 10, 2016

rickard-von-essen commented Oct 10, 2016

chris-smith-zocdoc commented Oct 10, 2016

bhcleek commented Oct 11, 2016

bhcleek commented Oct 14, 2016

bhcleek commented Oct 30, 2016

Holmistr commented Nov 21, 2016

bhcleek commented Nov 23, 2016 • edited Loading

Holmistr commented Nov 23, 2016

bhcleek commented Nov 26, 2016 • edited Loading

transamfire85 commented Dec 1, 2016

dragan commented Dec 5, 2016

dragan commented Dec 5, 2016

bhcleek commented Dec 6, 2016

dragan commented Dec 7, 2016

bhcleek commented Dec 7, 2016 • edited Loading

rickard-von-essen commented Sep 22, 2016 •

edited

Loading

bhcleek commented Sep 25, 2016 •

edited

Loading

bhcleek commented Sep 29, 2016 •

edited

Loading

rickard-von-essen commented Oct 4, 2016 •

edited

Loading

bhcleek commented Nov 23, 2016 •

edited

Loading

bhcleek commented Nov 26, 2016 •

edited

Loading

bhcleek commented Dec 7, 2016 •

edited

Loading