Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to push some refs #14

Open
Cryptophobia opened this issue Mar 20, 2018 · 19 comments
Open

Failed to push some refs #14

Cryptophobia opened this issue Mar 20, 2018 · 19 comments

Comments

@Cryptophobia
Copy link
Member

From @Overdrivr on June 14, 2017 6:39

I'm trying to deploy from Gitlab CI to Deis using buildpack ( I cannot use Docker-based because of #823 )

Authentication to deis from gitlab-ci runner works fine, code is built successfully. But at the end of the deployment, I get the following error message.

remote: Successfully built 76385f485fae        
remote: Pushing to registry        
remote: received unexpected HTTP status: 500 Internal Server Error        
remote: 2017/06/14 06:09:56 Error running git receive hook [Build pod exited with code 1, stopping build.]        
To ssh://git@deis-builder.35.187.115.255.nip.io:2222/cronobo-dispatcher.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@deis-builder.35.187.115.255.nip.io:2222/cronobo-dispatcher.git'
ERROR: Job failed: exit code 1

My cluster is GKE 3 node n1-standard1, with 50Go of storage each.

All pods are properly running

$ kubectl --namespace=deis get pods
NAME                                     READY     STATUS    RESTARTS   AGE
deis-builder-1134541883-4mvsv            1/1       Running   0          2h
deis-controller-2381495828-61b1t         1/1       Running   1          2h
deis-database-401209816-csdkz            1/1       Running   0          2h
deis-logger-2717637750-16mtl             1/1       Running   3          2h
deis-logger-fluentd-4n8b3                1/1       Running   0          2h
deis-logger-fluentd-86dsg                1/1       Running   0          2h
deis-logger-fluentd-x1w8f                1/1       Running   0          2h
deis-logger-redis-1413683677-vl4kd       1/1       Running   0          2h
deis-minio-3370481340-8nlfx              1/1       Running   0          2h
deis-monitor-grafana-1073006293-45rnw    1/1       Running   0          2h
deis-monitor-influxdb-2675149720-vmp7s   1/1       Running   0          2h
deis-monitor-telegraf-n9z5j              1/1       Running   1          2h
deis-monitor-telegraf-qhbtf              1/1       Running   1          2h
deis-monitor-telegraf-ttsgr              1/1       Running   1          2h
deis-nsqd-1225249577-c3jnk               1/1       Running   0          2h
deis-registry-2623437609-x1fcp           1/1       Running   1          2h
deis-registry-proxy-3mwz3                1/1       Running   0          2h
deis-registry-proxy-3pftk                1/1       Running   0          2h
deis-registry-proxy-ccqxp                1/1       Running   0          2h
deis-router-3258585802-fmfm9             1/1       Running   0          2h
deis-workflow-manager-3757950027-vkwn8   1/1       Running   0          2h

How can I debug this ?

Copied from original issue: deis/workflow#826
@Cryptophobia
Copy link
Member Author

From @dumyan on June 14, 2017 16:38

The push is rejected because of the Internal Server Error that happened probably when pushing the built image into registry (I'm not sure if a successful push to registry is logged). This is desired behavior because if the refs get pushed and the image cannot be successfully started in a container it will leave you with a bogus repository state which is not currently deployed. So, failed to push some refs is really your friend here and your enemies are probably hiding in the logs.

In short, you should inspect deis-builder and also deis-registry logs and work your way from there. You can also enable debug mode if you see nothing obvious in the logs.

@Cryptophobia
Copy link
Member Author

From @Overdrivr on June 15, 2017 6:1

Thanks for the reply. The registry pod if filled with info messages, is this the default behavior ?

10.12.0.1 - - [15/Jun/2017:05:53:12 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:12.050694806Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=fe682b3f-5756-47d9-8e86-cf19f5d1a9e3 http.request.method=GET http.request.remoteaddr="10.12.0.1:35296" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=8.927239ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0 
10.12.0.1 - - [15/Jun/2017:05:53:12 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:22.041218766Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=710364ac-dde4-4bb2-9723-cd2d383ca686 http.request.method=GET http.request.remoteaddr="10.12.0.1:35330" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.648287ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0 
10.12.0.1 - - [15/Jun/2017:05:53:22 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:22.04325269Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=80e1cbd3-52e5-405c-ab1c-928957f731de http.request.method=GET http.request.remoteaddr="10.12.0.1:35328" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.529342ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0

The deis-builder log is maybe more interesting

$ kubectl --namespace=deis logs deis-builder-1134541883-4mvsv 
2017/06/14 04:37:38 Starting health check server on port 8092
2017/06/14 04:37:38 Starting deleted app cleaner
2017/06/14 04:37:38 Starting SSH server on 0.0.0.0:2223
Listening on 0.0.0.0:2223
Accepting new connections.
---> ---> 2017/06/14 04:37:51 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:01 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:11 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:21 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:31 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:41 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:51 Readinesscheck endpoint timed out after 10s
2017/06/14 04:39:01 Readinesscheck endpoint timed out after 10s
Accepted connection.
Starting ssh authentication
Channel type: session

receiving git repo name: app-dispatcher.git, operation: git-receive-pack, fingerprint: 84:a8:7e:6a:e8:1b:20:34:4b:29:a7:f3:14:5a:e3:42, user: gitlabci
creating repo directory /home/git/app-dispatcher.git
writing pre-receive hook under /home/git/app-dispatcher.git
git-shell -c git-receive-pack 'app-dispatcher.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
Accepted connection.
Starting ssh authentication
Channel type: session

receiving git repo name: app-dispatcher.git, operation: git-receive-pack, fingerprint: 84:a8:7e:6a:e8:1b:20:34:4b:29:a7:f3:14:5a:e3:42, user: gitlabci
creating repo directory /home/git/app-dispatcher.git
writing pre-receive hook under /home/git/app-dispatcher.git
git-shell -c git-receive-pack 'app-dispatcher.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
Accepted connection.
---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> [ERROR] Failed handshake: EOF
Accepted connection.
Accepted connection.
---> ---> [ERROR] Failed handshake: EOF
[ERROR] Failed handshake: EOF
Accepted connection.
Accepted connection.
---> ---> [ERROR] Failed handshake: EOF
Accepted connection.
---> [ERROR] Failed handshake: EOF

Any idea what is causing the handshake to fail ?

@Cryptophobia
Copy link
Member Author

From @bacongobbler on June 15, 2017 6:2

This handshake failures are just the healthcheck probe. It's a red herring. What you want to be looking at is the registry logs as that's where the 500 error came from.

@Cryptophobia
Copy link
Member Author

From @Overdrivr on June 15, 2017 6:30

Here you go, registry logs with HTTP 500 error

$ kubectl --namespace=deis logs deis-registry-2623437609-x1fcp | grep " 500 "

10.12.0.7 - - [15/Jun/2017:06:15:15 +0000] "PUT /v2/app-dispatcher/blobs/uploads/a0a37674-7680-4e64-a809-05551e124f76?_state=9rUDYtxw_nxrKtVQVQK20EXSMhdleDG4Ci5BbZEZOjt7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiYTBhMzc2NzQtNzY4MC00ZTY0LWE4MDktMDU1NTFlMTI0Zjc2IiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNDozOFoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:20 +0000] "PUT /v2/app-dispatcher/blobs/uploads/05e4cd44-be93-459c-99a4-cf7d77f6519a?_state=omTy6YEGZqDnBs2fqLPOXwWbg6PTtpBzGAh8eVtsIMh7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMDVlNGNkNDQtYmU5My00NTljLTk5YTQtY2Y3ZDc3ZjY1MTlhIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNDozOVoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:54 +0000] "PUT /v2/app-dispatcher/blobs/uploads/f1cb1743-43c3-478b-a4b8-27c4ad3e4a8b?_state=xhjHl05OxmV09G4oskfU0CIC7gsbaczC5Evz1iX36jx7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZjFjYjE3NDMtNDNjMy00NzhiLWE0YjgtMjdjNGFkM2U0YThiIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNToyMVoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/d3fbfd1a-eb57-4cf8-a55b-fb26e0d38ac6?_state=qd0QZx4wanTPwiDZ7aVnXpr1vxR7fMZ_gHTEdmC8b1N7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZDNmYmZkMWEtZWI1Ny00Y2Y4LWE1NWItZmIyNmUwZDM4YWM2IiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNToyNloifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:56 +0000] "PUT /v2/app-dispatcher/blobs/uploads/2a7e85ff-768e-4407-b2e2-810261e0e5d7?_state=2mDRWSFFneGaJL4styC3IQ7Cd1bjnpw5FcOUZpiLUNB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMmE3ZTg1ZmYtNzY4ZS00NDA3LWIyZTItODEwMjYxZTBlNWQ3IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTQ6MzhaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:47 +0000] "PUT /v2/app-dispatcher/blobs/uploads/4213afc5-bd0c-4e89-8545-c7c62d97c973?_state=33N3ehczUMDazWB9sxMH7wN8ftYcLOL4hWyps1L_3P17Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNDIxM2FmYzUtYmQwYy00ZTg5LTg1NDUtYzdjNjJkOTdjOTczIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNjowNVoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:48 +0000] "PUT /v2/app-dispatcher/blobs/uploads/2670a319-80f6-4b31-9c60-8498fb2219c0?_state=wAgGDjU4XQc-K3tBivJ11HF0bi2rgxXNaocIxf5T9pB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMjY3MGEzMTktODBmNi00YjMxLTljNjAtODQ5OGZiMjIxOWMwIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNjowNloifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/46b17962-9a71-4e45-ae03-23b207184548?_state=aunuxweYQROhndJe4ZqAsmnCBLyWbq6r1P-DKY1p-nR7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNDZiMTc5NjItOWE3MS00ZTQ1LWFlMDMtMjNiMjA3MTg0NTQ4IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTY6MDJaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:36 +0000] "PUT /v2/app-dispatcher/blobs/uploads/3a644e41-d3c2-44a8-9596-ba42359aa71c?_state=1WfuVKBc2gc3qjb0siH7Lcbf2asHkHjdBDjYOh8T_ZB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiM2E2NDRlNDEtZDNjMi00NGE4LTk1OTYtYmE0MjM1OWFhNzFjIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzowMloifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:44 +0000] "PUT /v2/app-dispatcher/blobs/uploads/787c7538-1a4e-439c-ad51-a0658ef87e74?_state=b7y6Wvq1uz0pKW1PluUWMABckM3NtIRP6X9zu0IghcZ7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNzg3Yzc1MzgtMWE0ZS00MzljLWFkNTEtYTA2NThlZjg3ZTc0IiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzowNFoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/4b5c16e3-2c1e-4264-a6f4-8cc396affbac?_state=vM16auRY8mezk2pPhMIHKLIzgzVhjV43KRhxQihhYid7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNGI1YzE2ZTMtMmMxZS00MjY0LWE2ZjQtOGNjMzk2YWZmYmFjIiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTc6MDVaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:14 +0000] "PUT /v2/app-dispatcher/blobs/uploads/b3928bcf-36e5-4000-886c-7288fe3389c0?_state=Uhwuv4nSaQCI_MUn8Eeqc8q8OzO1KM74-_YwRZ7JER57Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiYjM5MjhiY2YtMzZlNS00MDAwLTg4NmMtNzI4OGZlMzM4OWMwIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzo1N1oifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:38 +0000] "PUT /v2/app-dispatcher/blobs/uploads/106c9553-86b6-4dfa-8b22-14bbf712b80f?_state=-5FSF76b4zMHCcWh2qKOfujnGVMTYZYp8dC7c73EFuB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMTA2Yzk1NTMtODZiNi00ZGZhLThiMjItMTRiYmY3MTJiODBmIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxODowNVoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:49 +0000] "PUT /v2/app-dispatcher/blobs/uploads/f5c80612-d1a4-44b7-9f1a-75a6096718ed?_state=BS_ZRX1Sv0kdnvr25YbNzOG8-f7jjJef2aTRlFKWf317Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZjVjODA2MTItZDFhNC00NGI3LTlmMWEtNzVhNjA5NjcxOGVkIiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTg6MTBaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:19:38 +0000] "PUT /v2/app-dispatcher/blobs/uploads/0592b1a6-5234-458d-9c8a-fa324a050517?_state=iNYDPE62ScgvCj825K8D-xthNqd_YffLVV2JfTzoQRd7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMDU5MmIxYTYtNTIzNC00NThkLTljOGEtZmEzMjRhMDUwNTE3IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTk6MDlaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"

@Cryptophobia
Copy link
Member Author

From @bacongobbler on June 15, 2017 6:37

Sounds like it's having a hard time uploading images to the backing storage driver as it's barking during PUT operations. I'd check your object storage logs and see if it's crashing, or perhaps your cluster is misconfigured and the registry is unable to upload blobs to object storage.

@Cryptophobia
Copy link
Member Author

From @Overdrivr on June 15, 2017 6:51

$ kubectl --namespace=deis logs deis-minio-3370481340-8nlfx 

minio server /home/minio/
2017/06/14 04:37:02 starting Minio server
2017/06/14 04:37:02 starting health check server on 0.0.0.0:8082
Migration from version ‘2’ to ‘3’ completed successfully.
Migration from version ‘3’ to ‘4’ completed successfully.

AccessKey: ****************  SecretKey: ********************  Region: us-east-1

Minio Object Storage:
    http://127.0.0.1:9000
    http://**.**.**.**:9000

Minio Browser:
    http://127.0.0.1:9000
    http://**.**.**.**:9000

To configure Minio Client:
    $ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
    $ chmod 755 mc
    $ ./mc config host add myminio http://127.0.0.1:9000 ****************   *************************

It's weird that minio is using region us-east-1, because my cluster is located in europe-west1-b. Could this be the problem ?

@Cryptophobia
Copy link
Member Author

From @bacongobbler on June 15, 2017 14:11

Minio fakes out and simulates that it's in us-east-1 to mimic the S3 API. It doesn't accurately reflect where it's deployed (in the cluster).

@Cryptophobia
Copy link
Member Author

From @EamonZhang on July 4, 2017 7:45

+1

@Cryptophobia
Copy link
Member Author

From @wikkid on August 7, 2017 15:12

Any updates on this? Got exactly the same problem.
I have bare metal kubernetes cluster and Deis workflow on top of that.
Sample applications are fine, but it doesn't work with my own.

@Cryptophobia
Copy link
Member Author

From @robeferre on August 8, 2017 16:55

Hi Guys any update on this Im having the same issue

@Cryptophobia
Copy link
Member Author

From @Overdrivr on August 9, 2017 4:25

I wasn't sure how to debug this further, so no updates so far. I wasn't seeing anything in my object storage logs.

@Cryptophobia
Copy link
Member Author

From @galbacarys on August 18, 2017 16:45

Having watched this carefully (I know, great way to debug things) I noticed something interesting-It looks like some of the minio requests are going through, but then the connection gets dropped somewhere (the object is too big?) and minio barfs and stops taking any more data.

[ deis registry logs ]
100.96.6.18 - - [18/Aug/2017:16:40:25 +0000] "POST /v2/aws-dx-selfjoin/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
time="2017-08-18T16:40:27.09674817Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="100.96.6.23:5000" http.request.id=cd874d81-d36b-48ec-8455-4528504f88f8 http.request.method=GET http.request.remoteaddr="100.96.6.1:53204" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" h
ttp.response.contenttype="application/json; charset=utf-8" http.response.duration=1.746968ms http.response.status=200 http.response.written=2 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.1 - - [18/Aug/2017:16:40:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-08-18T16:40:27.09757101Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="100.96.6.23:5000" http.request.id=72f0a983-963a-499e-89d6-d30c1b29fab7 http.request.method=GET http.request.remoteaddr="100.96.6.1:53202" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" h
ttp.response.contenttype="application/json; charset=utf-8" http.response.duration=1.58666ms http.response.status=200 http.response.written=2 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.1 - - [18/Aug/2017:16:40:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-08-18T16:40:35.222805315Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=6aaf4e8a-dc20-4dcb-a9da-9f2df76db945 http.request.method=PATCH http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-
9ab733dd225f?_state=0CLtxmCoH9cYWnDAEmjo9mWBMC3_Q7TX85-MY80Au4B7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDE2OjQwOjI1LjIxODg4NzY5MVoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4
.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.response.duration=1.380391587s http.response.status=202 http.response.written=0 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.18 - - [18/Aug/2017:16:40:33 +0000] "PATCH /v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=0CLtxmCoH9cYWnDAEmjo9mWBMC3_Q7TX85-MY80Au4B7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDE2
OjQwOjI1LjIxODg4NzY5MVoifQ%3D%3D HTTP/1.0" 202 0 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
time="2017-08-18T16:40:35.378123268Z" level=error msg="unknown error completing upload: s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: DZFQC2IC58RHYLEY" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3edb847-d84f-4394-b247-0d03d594ce4f http.request.method=
PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjA
xNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=regis
try vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version=v2.6.0
time="2017-08-18T16:40:35.378179784Z" level=error msg="error canceling upload after error: already closed" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3edb847-d84f-4394-b247-0d03d594ce4f http.request.method=PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/u
ploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f93
0bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version
=v2.6.0
time="2017-08-18T16:40:35.383839588Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: DZFQC2IC58RHYLEY" err.message="unknown error" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3
edb847-d84f-4394-b247-0d03d594ce4f http.request.method=PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMz
ZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.re
sponse.contenttype="application/json; charset=utf-8" http.response.duration=110.023913ms http.response.status=500 http.response.written=117 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version=v2.6.0
100.96.6.18 - - [18/Aug/2017:16:40:35 +0000] "PUT /v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC
0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845 HTTP/1.0" 500 117 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"

@Cryptophobia
Copy link
Member Author

From @dumyan on August 18, 2017 21:59

Hey, guys, you can try with a newer minio image - just edit the minio deployment. Rereading this made me vaguely remember that I had some issues with the minio version that deis deploys by default.

@Cryptophobia
Copy link
Member Author

From @battlemidget on August 19, 2017 20:16

@dumyan What new minio image are you referring to?

@Cryptophobia
Copy link
Member Author

From @dumyan on August 20, 2017 12:10

@battlemidget the OP is using minio as a storage backend for the registry. The default image for deis-minio is pretty outdated (or at least was outdated few months ago when I used minio as a backend storage). My suggestion is to change the minio image in the deis-minio deployment (you can try this image for example minio/minio:RELEASE.2017-07-24T18-27-35Z)

@Cryptophobia
Copy link
Member Author

From @battlemidget on August 20, 2017 20:21

@dumyan thanks im hitting this same issue, ill give that a try

@Cryptophobia
Copy link
Member Author

From @battlemidget on August 21, 2017 17:27

@dumyan Ok, so not exactly the same issue as this one it seems. I am using off cluster storage (s3) and our distribution of kubernetes uses flannel cni so I just needed to enable that.

FYI, we are heavily promoting Deis workflow as part of our core addons for our Canonical Distribution of Kubernetes, my first blog post on it is here http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/#addons, and the application is https://conjure-up.io. cc @bacongobbler

Thanks for the help!

@Cryptophobia
Copy link
Member Author

From @bacongobbler on August 21, 2017 18:52

@battlemidget you may want to read our latest blog post. https://deis.com/blog/2017/deis-workflow-final-release/

@Cryptophobia
Copy link
Member Author

From @battlemidget on August 21, 2017 18:58

@bacongobbler well that is disappointing :(

@Cryptophobia Cryptophobia mentioned this issue Mar 20, 2018
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Cryptophobia