Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker_network fails when trying to update network connected to swarm service #14

Open
kaysond opened this issue Aug 17, 2020 · 3 comments
Labels
docker-swarm Docker Swarm enhancement New feature or request

Comments

@kaysond
Copy link
Contributor

kaysond commented Aug 17, 2020

Migrated from ansible/ansible#65742

SUMMARY

docker_network doesn't check if a network is connected to any services before trying to remove it

ISSUE TYPE
  • Bug Report
COMPONENT NAME

docker_network

ANSIBLE VERSION
ansible 2.10.0.dev0
  config file = None
  configured module search path = [u'/home/administrator/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/ansible/lib/ansible
  executable location = /opt/ansible/bin/ansible
  python version = 2.7.15+ (default, Oct  7 2019, 17:39:04) [GCC 7.4.0]
CONFIGURATION
OS / ENVIRONMENT

Running ansible locally on Ubuntu 18.04

STEPS TO REPRODUCE

Create:
~/docker-compose.yml:

---
version: "3.7"

services:
  test:
    image: jwilder/whoami
    ports:
      - 8000:8000
    networks:
      - testnet

networks:
  testnet:
    external: true

playbook.yml:

---
- hosts: all
  connection: local
  tasks:
  - name: "create new swarm"
    docker_swarm:
      state: present

  - name: "create network"
    docker_network:
      name: "testnet"
      scope: "swarm"
      state: present

  - name: "deploy service"
    docker_stack:
      state: present
      name: "test"
      compose:
        - ~/docker-compose.yml

  - name: "update network ipam_config"
    docker_network:
      name: "testnet"
      scope: "swarm"
      internal: yes
      ipam_config:
        - subnet: 10.1.0.0/16
          gateway: 10.1.0.1
          iprange: 10.1.1.0/24
      state: present
      force: yes

Run ansible-playbook -i "127.0.0.1," -vvv ~/playbook.yml

EXPECTED RESULTS

testnet network should be recreated with the updated ipam_config.

Ideally, Ansible would also re-connect services based on the connected and append arguments, just as it would containers (I haven't tested this, but am guessing that because it does not appear to check services at all, the module will not do this)

ACTUAL RESULTS

The task chokes because the module attempts to remove the network while it is still connected to the test_test service.

The issue here is that services need to be updated to remove the network. The Docker command-line for this would be: docker service update --network-rm testnet test_test.


administrator@testserver:~$ ansible-playbook -i "127.0.0.1," -vvvv playbook.yml
ansible-playbook 2.10.0.dev0
  config file = None
  configured module search path = [u'/home/administrator/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/ansible/lib/ansible
  executable location = /opt/ansible/bin/ansible-playbook
  python version = 2.7.15+ (default, Oct  7 2019, 17:39:04) [GCC 7.4.0]
No config file found; using defaults
setting up inventory plugins
Set default localhost to 127.0.0.1
Parsed 127.0.0.1, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /opt/ansible/lib/ansible/plugins/callback/default.pyc

PLAYBOOK: playbook.yml *****************************************************************************************************************************************
Positional arguments: playbook.yml
become_method: sudo
inventory: (u'127.0.0.1,',)
forks: 5
tags: (u'all',)
verbosity: 4
connection: smart
timeout: 10
1 plays in playbook.yml

PLAY [all] *****************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************
task path: /home/administrator/playbook.yml:2
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: administrator
<127.0.0.1> EXEC /bin/sh -c 'echo ~administrator && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608 `" && echo ansible-tmp-1576094690.21-202394398444608="` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608 `" ) && sleep 0'
<127.0.0.1> Attempting python interpreter discovery
<127.0.0.1> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
Using module file /opt/ansible/lib/ansible/modules/system/setup.py
<127.0.0.1> PUT /home/administrator/.ansible/tmp/ansible-local-9291d3Y_9_/tmpbCDh_P TO /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608/AnsiballZ_setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608/ /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608/AnsiballZ_setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608/AnsiballZ_setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/administrator/.ansible/tmp/ansible-tmp-1576094690.21-202394398444608/ > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1]
META: ran handlers

TASK [create new swarm] ****************************************************************************************************************************************
task path: /home/administrator/playbook.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: administrator
<127.0.0.1> EXEC /bin/sh -c 'echo ~administrator && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793 `" && echo ansible-tmp-1576094691.99-165539105977793="` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793 `" ) && sleep 0'
Using module file /opt/ansible/lib/ansible/modules/cloud/docker/docker_swarm.py
<127.0.0.1> PUT /home/administrator/.ansible/tmp/ansible-local-9291d3Y_9_/tmp8BBdSG TO /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793/AnsiballZ_docker_swarm.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793/ /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793/AnsiballZ_docker_swarm.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793/AnsiballZ_docker_swarm.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/administrator/.ansible/tmp/ansible-tmp-1576094691.99-165539105977793/ > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1] => {
    "actions": [
        "No modification"
    ],
    "changed": false,
    "invocation": {
        "module_args": {
            "advertise_addr": null,
            "api_version": "auto",
            "autolock_managers": null,
            "ca_cert": null,
            "ca_force_rotate": null,
            "client_cert": null,
            "client_key": null,
            "debug": false,
            "default_addr_pool": null,
            "dispatcher_heartbeat_period": null,
            "docker_host": "unix://var/run/docker.sock",
            "election_tick": null,
            "force": false,
            "heartbeat_tick": null,
            "join_token": null,
            "keep_old_snapshots": null,
            "labels": null,
            "listen_addr": "0.0.0.0:2377",
            "log_entries_for_slow_followers": null,
            "name": null,
            "node_cert_expiry": null,
            "node_id": null,
            "remote_addrs": null,
            "rotate_manager_token": false,
            "rotate_worker_token": false,
            "signing_ca_cert": null,
            "signing_ca_key": null,
            "snapshot_interval": null,
            "ssl_version": null,
            "state": "present",
            "subnet_size": null,
            "task_history_retention_limit": null,
            "timeout": 60,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "result": "",
    "swarm_facts": {
        "CreatedAt": "2019-12-11T19:50:02.428917546Z",
        "DataPathPort": 4789,
        "DefaultAddrPool": [
            "10.0.0.0/8"
        ],
        "ID": "izh2k607cm14wxigl50uhi7je",
        "JoinTokens": {
            "Manager": "SWMTKN-1-3m709tij7cdul1uzrkbzpc4fih5qdv5uabfc8c0ebdbee0trg3-26ba25oq0murwhrq7i3se9dtv",
            "Worker": "SWMTKN-1-3m709tij7cdul1uzrkbzpc4fih5qdv5uabfc8c0ebdbee0trg3-4lqo4128vvnjmctlohec7pwud"
        },
        "RootRotationInProgress": false,
        "Spec": {
            "CAConfig": {
                "NodeCertExpiry": 7776000000000000
            },
            "Dispatcher": {
                "HeartbeatPeriod": 5000000000
            },
            "EncryptionConfig": {
                "AutoLockManagers": false
            },
            "Labels": {},
            "Name": "default",
            "Orchestration": {
                "TaskHistoryRetentionLimit": 5
            },
            "Raft": {
                "ElectionTick": 10,
                "HeartbeatTick": 1,
                "KeepOldSnapshots": 0,
                "LogEntriesForSlowFollowers": 500,
                "SnapshotInterval": 10000
            },
            "TaskDefaults": {}
        },
        "SubnetSize": 24,
        "TLSInfo": {
            "CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAElJ9nXDXrPNOeQHRw22HRCy+Njh6p4xJN100NXny4z6/FMHORBiySGAMwA3uxFT4APG6F4OpwInNe/wt3Yl4wQw==",
            "CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
            "TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBaTCCARCgAwIBAgIUGjRyDj4LrbBK8x9fUqWO+3eRVkYwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTkxMjExMTk0NTAwWhcNMzkxMjA2MTk0\nNTAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABJSfZ1w16zzTnkB0cNth0QsvjY4eqeMSTddNDV58uM+vxTBzkQYskhgDMAN7\nsRU+ADxuheDqcCJzXv8Ld2JeMEOjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBSDpOaG2zoLvCY+KtqaiT52VqClzjAKBggqhkjO\nPQQDAgNHADBEAiATVEjLDa81qyrD6p2hpMB6mlIJ8ARCRkXg+wNAZwNptQIgX1wJ\nf5u+cjlSEUkgWz2jQHchwB1i5WvczmBLMsJZYMY=\n-----END CERTIFICATE-----\n"
        },
        "UnlockKey": null,
        "UpdatedAt": "2019-12-11T19:50:02.552268297Z",
        "Version": {
            "Index": 10
        }
    }
}

TASK [create network] ******************************************************************************************************************************************
task path: /home/administrator/playbook.yml:9
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: administrator
<127.0.0.1> EXEC /bin/sh -c 'echo ~administrator && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453 `" && echo ansible-tmp-1576094693.07-94116159261453="` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453 `" ) && sleep 0'
Using module file /opt/ansible/lib/ansible/modules/cloud/docker/docker_network.py
<127.0.0.1> PUT /home/administrator/.ansible/tmp/ansible-local-9291d3Y_9_/tmpQ4cLzB TO /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453/AnsiballZ_docker_network.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453/ /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453/AnsiballZ_docker_network.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453/AnsiballZ_docker_network.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/administrator/.ansible/tmp/ansible-tmp-1576094693.07-94116159261453/ > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1] => {
    "ansible_facts": {
        "docker_network": {
            "Attachable": false,
            "ConfigFrom": {
                "Network": ""
            },
            "ConfigOnly": false,
            "Containers": {},
            "Created": "2019-12-11T19:56:35.685260011Z",
            "Driver": "bridge",
            "EnableIPv6": false,
            "IPAM": {
                "Config": [
                    {
                        "Gateway": "172.19.0.1",
                        "Subnet": "172.19.0.0/16"
                    }
                ],
                "Driver": "default",
                "Options": null
            },
            "Id": "ydi7ktuaxvyvf17xvs4kwcv6h",
            "Ingress": false,
            "Internal": false,
            "Labels": {},
            "Name": "testnet",
            "Options": {},
            "Scope": "swarm"
        }
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "appends": false,
            "attachable": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "connected": [],
            "debug": false,
            "docker_host": "unix://var/run/docker.sock",
            "driver": "bridge",
            "driver_options": {},
            "enable_ipv6": null,
            "force": false,
            "internal": null,
            "ipam_config": null,
            "ipam_driver": null,
            "ipam_driver_options": null,
            "ipam_options": {
                "aux_addresses": null,
                "gateway": null,
                "iprange": null,
                "subnet": null
            },
            "labels": {},
            "name": "testnet",
            "scope": "swarm",
            "ssl_version": null,
            "state": "present",
            "timeout": 60,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "network": {
        "Attachable": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Created": "2019-12-11T19:56:35.685260011Z",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Config": [
                {
                    "Gateway": "172.19.0.1",
                    "Subnet": "172.19.0.0/16"
                }
            ],
            "Driver": "default",
            "Options": null
        },
        "Id": "ydi7ktuaxvyvf17xvs4kwcv6h",
        "Ingress": false,
        "Internal": false,
        "Labels": {},
        "Name": "testnet",
        "Options": {},
        "Scope": "swarm"
    }
}

TASK [deploy service] ******************************************************************************************************************************************
task path: /home/administrator/playbook.yml:15
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: administrator
<127.0.0.1> EXEC /bin/sh -c 'echo ~administrator && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851 `" && echo ansible-tmp-1576094694.11-236202939741851="` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851 `" ) && sleep 0'
Using module file /opt/ansible/lib/ansible/modules/cloud/docker/docker_stack.py
<127.0.0.1> PUT /home/administrator/.ansible/tmp/ansible-local-9291d3Y_9_/tmpPDRyQg TO /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851/AnsiballZ_docker_stack.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851/ /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851/AnsiballZ_docker_stack.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851/AnsiballZ_docker_stack.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/administrator/.ansible/tmp/ansible-tmp-1576094694.11-236202939741851/ > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "absent_retries": 0,
            "absent_retries_interval": 1,
            "compose": [
                "~/docker-compose.yml"
            ],
            "name": "test",
            "prune": false,
            "resolve_image": null,
            "state": "present",
            "with_registry_auth": false
        }
    },
    "rc": 0,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "Updating service test_test (id: bdkua642raovsbpl97cpi240z)\n",
    "stdout_lines": [
        "Updating service test_test (id: bdkua642raovsbpl97cpi240z)"
    ]
}

TASK [update network ipam_config] ******************************************************************************************************************************
task path: /home/administrator/playbook.yml:22
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: administrator
<127.0.0.1> EXEC /bin/sh -c 'echo ~administrator && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305 `" && echo ansible-tmp-1576094697.66-59738835547305="` echo /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305 `" ) && sleep 0'
Using module file /opt/ansible/lib/ansible/modules/cloud/docker/docker_network.py
<127.0.0.1> PUT /home/administrator/.ansible/tmp/ansible-local-9291d3Y_9_/tmpti_GlQ TO /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305/AnsiballZ_docker_network.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305/ /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305/AnsiballZ_docker_network.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305/AnsiballZ_docker_network.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/administrator/.ansible/tmp/ansible-tmp-1576094697.66-59738835547305/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_docker_network_payload_iy71EX/ansible_docker_network_payload.zip/ansible/modules/cloud/docker/docker_network.py", line 692, in main
  File "/tmp/ansible_docker_network_payload_iy71EX/ansible_docker_network_payload.zip/ansible/modules/cloud/docker/docker_network.py", line 396, in __init__
  File "/tmp/ansible_docker_network_payload_iy71EX/ansible_docker_network_payload.zip/ansible/modules/cloud/docker/docker_network.py", line 615, in present
  File "/tmp/ansible_docker_network_payload_iy71EX/ansible_docker_network_payload.zip/ansible/modules/cloud/docker/docker_network.py", line 560, in remove_network
  File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/network.py", line 187, in remove_network
    self._raise_for_status(res)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
APIError: 400 Client Error: Bad Request ("rpc error: code = FailedPrecondition desc = network ydi7ktuaxvyvf17xvs4kwcv6h is in use by service bdkua642raovsbpl97cpi240z")

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "appends": false,
            "attachable": null,
            "ca_cert": null,
            "client_cert": null,
            "client_key": null,
            "connected": [],
            "debug": false,
            "docker_host": "unix://var/run/docker.sock",
            "driver": "bridge",
            "driver_options": {},
            "enable_ipv6": null,
            "force": true,
            "internal": true,
            "ipam_config": [
                {
                    "aux_addresses": null,
                    "gateway": "10.1.0.1",
                    "iprange": "10.1.1.0/24",
                    "subnet": "10.1.0.0/16"
                }
            ],
            "ipam_driver": null,
            "ipam_driver_options": null,
            "ipam_options": {
                "aux_addresses": null,
                "gateway": null,
                "iprange": null,
                "subnet": null
            },
            "labels": {},
            "name": "testnet",
            "scope": "swarm",
            "ssl_version": null,
            "state": "present",
            "timeout": 60,
            "tls": false,
            "tls_hostname": "localhost",
            "validate_certs": false
        }
    },
    "msg": "An unexpected docker error occurred: 400 Client Error: Bad Request (\"rpc error: code = FailedPrecondition desc = network ydi7ktuaxvyvf17xvs4kwcv6h is in use by service bdkua642raovsbpl97cpi240z\")"
}

PLAY RECAP *****************************************************************************************************************************************************
127.0.0.1                  : ok=4    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
@ansibullbot
Copy link

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@felixfontein
Copy link
Collaborator

For anyone interested in why this hasn't happened yet and what is necessary to implement this, see the discussion in ansible/ansible#65742.

@gundalow gundalow transferred this issue from ansible-collections/community.general Oct 30, 2020
@felixfontein felixfontein added the docker-swarm Docker Swarm label Jan 2, 2021
@felixfontein felixfontein added the enhancement New feature or request label Sep 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docker-swarm Docker Swarm enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants