Behind a proxy: Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build: #465

maizumi · 2019-02-26T10:14:23Z

[root@ENVT_Lab_CentOS1 kafka-docker]# docker-compose up
Building kafka
Step 1/14 : FROM openjdk:8u191-jre-alpine
 ---> 1b46cc2ba839
Step 2/14 : ARG kafka_version=2.1.1
 ---> Using cache
 ---> e0822b244bf1
Step 3/14 : ARG scala_version=2.12
 ---> Using cache
 ---> 1a2298c6f6f3
Step 4/14 : ARG glibc_version=2.29-r0
 ---> Using cache
 ---> d40c36f12c93
Step 5/14 : ARG vcs_ref=unspecified
 ---> Using cache
 ---> e2705fab268d
Step 6/14 : ARG build_date=unspecified
 ---> Using cache
 ---> 34f382b77219
Step 7/14 : LABEL org.label-schema.name "kafka" org.label-schema.description "Apache Kafka" org.label-schema.build-date "${build_date}" org.label-schema.vcs-url "https://github.com/wurstmeister/kafka-docker" org.label-schema.vcs-ref "${vcs_ref}" org.label-schema.version "${scala_version}_${kafka_version}" org.label-schema.schema-version "1.0" maintainer "wurstmeister"
 ---> Using cache
 ---> b654b59b70e2
Step 8/14 : ENV KAFKA_VERSION $kafka_version SCALA_VERSION $scala_version KAFKA_HOME /opt/kafka GLIBC_VERSION $glibc_version
 ---> Using cache
 ---> 6324b68895ed
Step 9/14 : ENV PATH ${PATH}:${KAFKA_HOME}/bin
 ---> Using cache
 ---> a4614fbd0803
Step 10/14 : COPY download-kafka.sh start-kafka.sh broker-list.sh create-topics.sh versions.sh /tmp/
 ---> Using cache
 ---> 9a51479887a3
Step 11/14 : RUN apk add --no-cache bash curl jq docker  && chmod a+x /tmp/*.sh  && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin  && sync && /tmp/download-kafka.sh  && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt  && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz  && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka  && rm /tmp/*  && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk  && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk  && rm glibc-${GLIBC_VERSION}.apk
 ---> Running in f5bb101f1d5f

fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/22) Installing ncurses-terminfo-base (6.1_p20190105-r0)
(2/22) Installing ncurses-terminfo (6.1_p20190105-r0)
(3/22) Installing ncurses-libs (6.1_p20190105-r0)
(4/22) Installing readline (7.0.003-r1)
(5/22) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
(6/22) Installing nghttp2-libs (1.35.1-r0)
(7/22) Installing libssh2 (1.8.0-r4)
(8/22) Installing libcurl (7.63.0-r0)
(9/22) Installing curl (7.63.0-r0)
(10/22) Installing libseccomp (2.3.3-r1)
(11/22) Installing runc (1.0.0_rc6-r1)
(12/22) Installing containerd (1.2.2-r0)
(13/22) Installing libmnl (1.0.4-r0)
(14/22) Installing jansson (2.11-r0)
(15/22) Installing libnftnl-libs (1.1.1-r0)
(16/22) Installing iptables (1.6.2-r1)
(17/22) Installing tini-static (0.18.0-r0)
(18/22) Installing device-mapper-libs (2.02.182-r0)
(19/22) Installing libltdl (2.4.6-r5)
(20/22) Installing docker (18.09.1-r0)
Executing docker-18.09.1-r0.pre-install
(21/22) Installing oniguruma (6.9.1-r0)
(22/22) Installing jq (1.6-r0)
Executing busybox-1.29.3-r10.trigger
OK: 349 MiB in 75 packages
Downloading kafka 2.1
Unable to determine mirror for downloading Kafka, the service may be down
ERROR: Service 'kafka' failed to build: The command '/bin/sh -c apk add --no-cache bash curl jq docker  && chmod a+x /tmp/*.sh  && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin  && sync && /tmp/download-kafka.sh  && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt  && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz  && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka  && rm /tmp/*  && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk  && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk  && rm glibc-${GLIBC_VERSION}.apk' returned a non-zero code: 1
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

The text was updated successfully, but these errors were encountered:

jaceklaskowski · 2019-02-26T10:40:07Z

Looks like /tmp/download-kafka.sh failed to download kafka 2.1 due to:

Unable to determine mirror for downloading Kafka, the service may be down

maizumi · 2019-02-26T10:50:59Z

Do you have any idea why it failed to download? I looked at download-kafka.sh but no idea what's going on.

jaceklaskowski · 2019-02-26T12:41:44Z

Unfortunately not. I'm sure you won't wait long before @sscaling helps you out.

sscaling · 2019-02-26T15:18:09Z

Most likely; either 2.1.1 hasn't propagated to your local mirrors yet or you're behind a firewall/proxy stopping you access the apache closer.cgi script.

Have a look at the raw output from the service

$ curl "https://www.apache.org/dyn/closer.cgi?path=/kafka/2.1.1/kafka_2.12-2.1.1.tgz&as_json=1"
{
       "backup": [ "https://www-eu.apache.org/dist/", "https://www-us.apache.org/dist/" ],
         "cca2": "nl",
          "ftp": [ "ftp://apache.proserve.nl/apache/", "ftp://mirror1.spango.com/apache/" ],
         "http": [ "http://apache.40b.nl/", "http://apache.cs.uu.nl/", "http://apache.hippo.nl/", "http://apache.mirror.h1.nl/", "http://apache.mirror.triple-it.nl/", "http://apache.mirror1.spango.com/", "http://apache.proserve.nl/", "http://apache.redkiwi.nl/", "http://ftp.nluug.nl/internet/apache/", "http://ftp.tudelft.nl/apache/", "http://mirror.koddos.net/apache/", "http://mirror.novg.net/apache/", "http://mirrors.supportex.net/apache/" ],
     "in_attic": false,
      "in_dist": true,
    "path_info": "kafka/2.1.1/kafka_2.12-2.1.1.tgz",
    "preferred": "http://ftp.nluug.nl/internet/apache/"
}

In the above example, if you visit http://ftp.nluug.nl/internet/apache/kafka/ you can verify that the version is there. Because the service suggests mirrors based on your egress IP's geographic location you may be getting results for mirrors that haven't been updated with the latest 2.1.1 code yet.

Try rebuilding (the preferred mirrors are rotated to distribute requests). You can verify this by calling the above curl command multiple times.

maizumi · 2019-02-26T23:07:58Z

My server is behind firewall/proxy but I check both and it should be ok(checked dns query result with nslookup and accessibility with curl and wget.)

[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# curl "https://www.apache.org/dyn/closer.cgi?path=/kafka/2.1.1/kafka_2.12-2.1.1.tgz&as_json=1"
{
       "backup": [ "https://www-eu.apache.org/dist/", "https://www-us.apache.org/dist/" ],
         "cca2": "us",
          "ftp": [ "ftp://apache.cs.utah.edu/apache.org/", "ftp://apache.mirrors.tds.net/pub/apache.org/", "ftp://ftp.osuosl.org/pub/apache/", "ftp://mirror.reverse.net/pub/apache/" ],
         "http": [ "http://apache.claz.org/", "http://apache.cs.utah.edu/", "http://apache.mirrors.hoobly.com/", "http://apache.mirrors.ionfish.org/", "http://apache.mirrors.lucidnetworks.net/", "http://apache.mirrors.pair.com/", "http://apache.mirrors.tds.net/", "http://apache.osuosl.org/", "http://apache.spinellicreations.com/", "http://ftp.naz.com/apache/", "http://ftp.wayne.edu/apache/", "http://mirror.cc.columbia.edu/pub/software/apache/", "http://mirror.cogentco.com/pub/apache/", "http://mirror.metrocast.net/apache/", "http://mirror.olnevhost.net/pub/apache/", "http://mirror.reverse.net/pub/apache/", "http://mirrors.advancedhosters.com/apache/", "http://mirrors.gigenet.com/apache/", "http://mirrors.ibiblio.org/apache/", "http://mirrors.koehn.com/apache/", "http://mirrors.ocf.berkeley.edu/apache/", "http://mirrors.sonic.net/apache/", "http://us.mirrors.quenda.co/apache/", "http://www.trieuvan.com/apache/" ],
     "in_attic": false,
      "in_dist": true,
    "path_info": "kafka/2.1.1/kafka_2.12-2.1.1.tgz",
    "preferred": "http://mirror.olnevhost.net/pub/apache/"
}[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# curl "https://www.apache.org/dyn/closer.cgi?path=/kafka/2.1.1/kafka_2.12-2.1.1.tgz&as_json=1"
{
       "backup": [ "https://www-eu.apache.org/dist/", "https://www-us.apache.org/dist/" ],
         "cca2": "us",
          "ftp": [ "ftp://apache.cs.utah.edu/apache.org/", "ftp://apache.mirrors.tds.net/pub/apache.org/", "ftp://ftp.osuosl.org/pub/apache/", "ftp://mirror.reverse.net/pub/apache/" ],
         "http": [ "http://apache.claz.org/", "http://apache.cs.utah.edu/", "http://apache.mirrors.hoobly.com/", "http://apache.mirrors.ionfish.org/", "http://apache.mirrors.lucidnetworks.net/", "http://apache.mirrors.pair.com/", "http://apache.mirrors.tds.net/", "http://apache.osuosl.org/", "http://apache.spinellicreations.com/", "http://ftp.naz.com/apache/", "http://ftp.wayne.edu/apache/", "http://mirror.cc.columbia.edu/pub/software/apache/", "http://mirror.cogentco.com/pub/apache/", "http://mirror.metrocast.net/apache/", "http://mirror.olnevhost.net/pub/apache/", "http://mirror.reverse.net/pub/apache/", "http://mirrors.advancedhosters.com/apache/", "http://mirrors.gigenet.com/apache/", "http://mirrors.ibiblio.org/apache/", "http://mirrors.koehn.com/apache/", "http://mirrors.ocf.berkeley.edu/apache/", "http://mirrors.sonic.net/apache/", "http://us.mirrors.quenda.co/apache/", "http://www.gtlib.gatech.edu/pub/apache/", "http://www.trieuvan.com/apache/" ],
     "in_attic": false,
      "in_dist": true,
    "path_info": "kafka/2.1.1/kafka_2.12-2.1.1.tgz",
    "preferred": "http://mirrors.sonic.net/apache/"
}[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# nslookup mirror.olnevhost.net
Server:		171.70.168.183
Address:	171.70.168.183#53

Non-authoritative answer:
Name:	mirror.olnevhost.net
Address: 188.165.227.148

[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# nslookup mirrors.sonic.net
Server:		171.70.168.183
Address:	171.70.168.183#53

Non-authoritative answer:
mirrors.sonic.net	canonical name = mirrors.sjc.sonic.net.
Name:	mirrors.sjc.sonic.net
Address: 157.131.0.16

[root@ENVT_Lab_CentOS1 kafka-docker]# 
[root@ENVT_Lab_CentOS1 kafka-docker]# wget http://mirror.olnevhost.net/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz
--2019-02-27 08:01:08--  http://mirror.olnevhost.net/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz
Connecting to 173.36.240.172:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: 55351618 (53M) [application/x-gzip]
Saving to: 'kafka_2.12-2.1.1.tgz.2'

 2% [==>                                                                                                                                   ] 1,507,904    283KB/s  eta 3m 18s ^C
[root@ENVT_Lab_CentOS1 kafka-docker]# wget http://mirrors.sonic.net/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz
--2019-02-27 08:01:28--  http://mirrors.sonic.net/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz
Connecting to 173.36.240.172:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: 55351618 (53M) [application/x-gzip]
Saving to: 'kafka_2.12-2.1.1.tgz.3'

 1% [=>                                                                                                                                    ] 1,079,327    269KB/s  eta 3m 25s ^C
[root@ENVT_Lab_CentOS1 kafka-docker]#

However "ftp" doesn't work with proxy. I need to check why but do you think this is cause of error?

sscaling · 2019-02-27T10:32:01Z

We use wget to download the binary over http:// so FTP shouldn't be the problem. I would suggest investigating from inside the container, it's possible wget isn't obeying any proxy config set, try something like

without proxy:

docker run --rm -it  openjdk:8u191-jre-alpine /bin/sh -c "wget  http://mirror.olnevhost.net/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz"

with proxy:

docker run --rm -it  openjdk:8u191-jre-alpine /bin/sh -c "wget -Y on http://mirror.olnevhost.net/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz"

You may need to pass through your HTTP_PROXY / http_proxy to the container environment (i.e. -e HTTP_PROXY="http://173.36.240.172") for the -Y on option to work.

maizumi · 2019-02-27T12:18:55Z

docker run --rm -it  openjdk:8u191-jre-alpine /bin/sh -c "wget -Y on http://mirror.olnevhost.net/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz"

This does work.

[root@ENVT_Lab_CentOS1 kafka-docker]# docker run --rm -it openjdk:8u191-jre-alpine /bin/sh -c "wget -Y on http://mirror.olnevhost.net/pub/apache/kafka/2.1.1/kafka_2.12-2.1.1.tgz"

Connecting to mirror.olnevhost.net (188.165.227.148:80)
kafka_2.12-2.1.1.tgz 100% |******************************************************************************************************************| 52.7M 0:00:00 ETA
[root@ENVT_Lab_CentOS1 kafka-docker]#

Which file should I fix with "-Y on"?

sscaling · 2019-02-27T13:38:23Z

Apologies - the issue was with contacting the closer.cgi service, so you'll need to configure the proxy for curl. You should be able to pass through the https_proxy environment var. You can still verify the proxy is the issue with wget though, something like:

docker run --rm -it openjdk:8u191-jre-alpine /bin/sh -c ''wget -O - -Y on "https://www.apache.org/dyn/closer.cgi?path=/kafka/2.1.1/kafka_2.12-2.1.1.tgz&as_json=1"'

If this works, you'll need to specify the https_proxy environment var for curl usage, i.e.

To do this, I think that you'll need to pass in the http_proxy so that it is available to the build context, by adding an arg to the Dockerfile

ARG https_proxy
ENV https_proxy ${https_proxy}

Then you should be able to run the docker build with docker build --build-arg "https_proxy=http://173.36.240.172" -t wurstmeister/kafka .

Typically I think if you are just using the image as is, people tend to mirror the pre-built image to their local/accessible docker repository or build in a CI system and push there.

maizumi · 2019-03-01T16:22:58Z

Thanks for your comment.
I actually gave up doing with proxy. now everything is good to go.

sscaling · 2019-03-03T20:43:47Z

OK, will close. Thanks.

BaronSam3di · 2021-02-02T13:28:04Z

😞 This link http://ftp.wayne.edu/apache/kafka/2.3.1/kafka_2.11-2.3.1.tgz is dead.

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /apache/kafka/2.3.1/kafka_2.11-2.3.1.tgz was not found on this server.</p>
</body></html>

I did find an alternative that seems legit https://archive.apache.org/dist/kafka/2.3.1/kafka-2.3.1-src.tgz

IS this the right place to ask for it to be swapped? 😌

Thanks

sscaling closed this as completed Mar 3, 2019

sscaling changed the title ~~Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build:~~ Behind a proxy: Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build: Mar 3, 2019

sscaling mentioned this issue Mar 19, 2019

DNS lookup error while running docker-compose up -d #471

Closed

sscaling mentioned this issue Apr 16, 2019

BusyBox wget does not correctly handle SSL over HTTPS proxy #487

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Behind a proxy: Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build: #465

Behind a proxy: Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build: #465

maizumi commented Feb 26, 2019 •

edited by sscaling

Loading

jaceklaskowski commented Feb 26, 2019

maizumi commented Feb 26, 2019

jaceklaskowski commented Feb 26, 2019

sscaling commented Feb 26, 2019

maizumi commented Feb 26, 2019 •

edited by sscaling

Loading

sscaling commented Feb 27, 2019

maizumi commented Feb 27, 2019

sscaling commented Feb 27, 2019

maizumi commented Mar 1, 2019

sscaling commented Mar 3, 2019

BaronSam3di commented Feb 2, 2021

Behind a proxy: Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build: #465

Behind a proxy: Unable to determine mirror for downloading Kafka, the service may be down ERROR: Service 'kafka' failed to build: #465

Comments

maizumi commented Feb 26, 2019 • edited by sscaling Loading

jaceklaskowski commented Feb 26, 2019

maizumi commented Feb 26, 2019

jaceklaskowski commented Feb 26, 2019

sscaling commented Feb 26, 2019

maizumi commented Feb 26, 2019 • edited by sscaling Loading

sscaling commented Feb 27, 2019

maizumi commented Feb 27, 2019

sscaling commented Feb 27, 2019

maizumi commented Mar 1, 2019

sscaling commented Mar 3, 2019

BaronSam3di commented Feb 2, 2021

maizumi commented Feb 26, 2019 •

edited by sscaling

Loading

maizumi commented Feb 26, 2019 •

edited by sscaling

Loading