17.06 won't deploy stack anymore. no suitable node (unsupported platform on 3 nodes) on armhf docker cluster

[trunet]

Hello,

I have an armhf cluster and since 17.06 installed, I'm getting this error and my stacks doesn't come up, it stays as pending forever. There's no constraint. This was working perfectly on 17.03.

Inspecting the task, I'm getting:

        "Status": {
            "Timestamp": "2017-07-02T12:51:00.556959128Z",
            "State": "pending",
            "Message": "no suitable node (unsupported platform on 3 nodes)",
            "ContainerStatus": {},
            "PortStatus": {}
        },

docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 17.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local nfs
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: chzqrk30d8aph7ikg60owdbjz
 Is Manager: true
 ClusterID: 1ucdfzovu4whdawkzv8wbfeb6
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 192.168.178.6
 Manager Addresses:
  192.168.178.6:2377
  192.168.178.7:2377
  192.168.178.8:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.34-45
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: armv7l
CPUs: 8
Total Memory: 1.949GiB
Name: odroid01.casaams.wsartori.com
ID: DRWB:PHMO:GCXF:GDTK:QXDF:V2WM:6ZAM:MC5G:GYID:7YOH:PBQC:BOYE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 registry:5000
 127.0.0.0/8
Live Restore Enabled: false

[trunet]

probably introduced on 8edbb92 by @nishanttotla

[nishanttotla]

@trunet can you show the output for service inspect and also what docker info on the nodes looks like?

[trunet]

docker info is on the first comment.

docker service inspect:

[
    {
        "ID": "n0bhpy9ibyiiv2wkux65x1dsv",
        "Version": {
            "Index": 2286
        },
        "CreatedAt": "2017-07-02T21:13:59.560944718Z",
        "UpdatedAt": "2017-07-02T21:13:59.56740151Z",
        "Spec": {
            "Name": "portal_portal",
            "Labels": {
                "com.docker.stack.image": "arm32v7/httpd:latest",
                "com.docker.stack.namespace": "portal"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "arm32v7/httpd:latest@sha256:90fb0d45f7267ac96523f2030ab842d61c77c65999024b5f67771f599ae4bfa8",
                    "Labels": {
                        "com.docker.stack.namespace": "portal"
                    },
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/mnt/glusterfs/portal",
                            "Target": "/usr/local/apache2/htdocs"
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "arm",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "jzho3a68yjsqvakajo7b9yqv3",
                        "Aliases": [
                            "portal"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 2
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 80,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "sposg3gl89sjhlhy1hdpqbwt6",
                    "Addr": "10.255.0.8/16"
                },
                {
                    "NetworkID": "jzho3a68yjsqvakajo7b9yqv3",
                    "Addr": "10.0.1.2/24"
                }
            ]
        }
    }
]

[nishanttotla]

I see the issue.

It seems like the node reports

OSType: linux
Architecture: armv7l

but the image reports

OSType: linux
Architecture: arm

and that causes scheduling to fail. We will need to add more normalization for architecture names. I'll do that but it would be useful to have a full list of architecture name variations.

cc @thaJeztah @aaronlehmann currently we only do this for x86_64 and amd64: https://github.com/docker/swarmkit/blob/master/manager/scheduler/filter.go#L288

[trunet]

I can think of aarch64 armv7l, but they are completely different architecture, aarch64 is 64-bits, armv7l is 32-bits.

also, check this: https://github.com/docker-library/official-images#architectures-other-than-amd64

[trunet]

@nishanttotla Is there a way to build the image with armv7l architecture instead of only arm, I didn't find anything on commit, push on build to add this? how docker hub is handling that, I suppose all https://hub.docker.com/r/arm32v7/ should uses armv7l?

[nishanttotla]

@trunet I'm not sure about that, but we'll see what's the best way to resolve this. Until then, you can get around this issue by using the --no-resolve-image flag on service create/update which will not add platform information to your service spec.

[trunet]

@nicolaka as I'm using stack, I didn't find the --no-resolve-image equivalent on compose 3.3 reference.

[trunet]

@nicolaka found it on https://docs.docker.com/engine/reference/commandline/stack_deploy/ thanks man

[nishanttotla]

@trunet apologies, yes for stacks you can use --resolve-image=never for now.

[thaJeztah]

We will need to add more normalization for architecture names.

Is there a list of valid architectures in the image's data? (Same for nodes)?

ARM is specifically tricky, as there's a lot of variations, and ways to express them (armhf, armv6, armv7 etc.)

If there's a list of valid values, we can decide to ignore other values

[nishanttotla]

Related: opencontainers/image-spec#661

@trunet I'm working on figuring out what the right fix is for this case.

[stevvooe]

@tianon Do you know how Config.Architecture is getting set for images in arm32v7? Are these just using GOARCH directly or is there a special build process?