Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package recently broken on ubuntu 16.04 ? #819

Closed
taynaud opened this issue Mar 19, 2018 · 11 comments
Closed

Package recently broken on ubuntu 16.04 ? #819

taynaud opened this issue Mar 19, 2018 · 11 comments

Comments

@taynaud
Copy link

taynaud commented Mar 19, 2018

Hello,

On an ubuntu 16.04, when I install the latest tensorflow serving package, it does not start with:

~$ tensorflow_model_server --help
tensorflow_model_server: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by tensorflow_model_server)
tensorflow_model_server: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by tensorflow_model_server)

Idem when I really try to serve models.

My system is up to date.

It was working perfectly before an apt update + upgrade this week end (~03/17, not sure of exact date).

Current package state:

~(1)$ apt show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.6.0
Built-Using: Bazel
Priority: optional
Section: contrib/devel
Maintainer: TensorFlow Serving team
Installed-Size: inconnu
Homepage: https://github.com/tensorflow/serving
Download-Size: 88,8 MB
APT-Manual-Installed: yes
APT-Sources: http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages
Description: TensorFlow Serving ModelServer
~$ apt-cache show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.6.0
Architecture: all
Maintainer: TensorFlow Serving team
Priority: optional
Section: contrib/devel
Filename: pool/tensorflow-model-server/t/tensorflow-model-server/tensorflow-model-server_1.6.0_all.deb
Size: 88812004
SHA256: 04f7e0937ffd6554e04d95eb850a7740ad81ad3c39d594b4d213dc6b14d775d7
SHA1: 4bc7445661aeacc03fa373474c3be4ba86eee8dd
MD5sum: ceabe0a8fa3a3809015c5cd203714488
Description: TensorFlow Serving ModelServer
Description-md5: 9b7e03f5296f318009581d6e285e2f89
Homepage: https://github.com/tensorflow/serving
Built-Using: Bazel
~$ uname -a
Linux Lenovo 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Do you have any clue ? Is it an issue with the package or do I need another dependency ?

Best,

@sbadia
Copy link

sbadia commented Mar 19, 2018

Hi!

Same issue here with Ubuntu LTS 16.04, the 1.5.0 package is not listed anymore in the Release file, but you can download it via a wget.

root@0a96112b4fac:~# file 1.5.0/usr/bin/tensorflow_model_server 
1.5.0/usr/bin/tensorflow_model_server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=d644ec743c5c224deea8fbad3378d9e0733769d4, not stripped
root@0a96112b4fac:~# file 1.6.0/usr/bin/tensorflow_model_server 
1.6.0/usr/bin/tensorflow_model_server: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=22568349c0aa5e934bb606ac117af06f4bc41be3, not stripped

@klockeph
Copy link

From how I understood several AskUbuntu and Stackoverflow threads, you have to enable the newest gcc version, by adding a certain repository to your apt-get package sources.

apt-get install -y software-properties-common # if not already installed
add-apt-repository ppa:ubuntu-toolchain-r/test -y

And then do the following commands (until it works, I had to do a dist-upgrade, but it could work earlier if I understood it correctly)

apt-get update
apt-get upgrade
apt-get dist-upgrade

Note that I usually don't use Ubuntu/apt-get and only did this to fix my docker.

Credits to https://askubuntu.com/a/582910

@lc0
Copy link

lc0 commented Mar 20, 2018

Seems like we would need to update https://www.tensorflow.org/serving/setup#installing_using_apt-get

@rsepassi
Copy link

Seeing the same. Here's a Travis build that uses the model server with error logs:
https://travis-ci.org/rsepassi/tensor2tensor/jobs/357041726#L2254

@nuchi
Copy link

nuchi commented Mar 26, 2018

I had the same problem, and it worked perfectly when I used the ubuntu:18.04 docker image instead of 16.04.

@dcyoung
Copy link

dcyoung commented Mar 26, 2018

As @sbadia mentioned above, pulling the packages directly seemed to work. Here is an updated dockerfile that worked for me.

FROM ubuntu:16.04
# Install general packages
RUN apt-get update && apt-get install -y \
        curl \
        libcurl3-dev \
        unzip \
        wget \
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Previous Installation of tensorflow-model-server (BROKEN RECENTLY)
#RUN echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list \
#    && curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add - \
#    && apt-get update && apt-get install tensorflow-model-server

# New installation of tensorflow-model-server
RUN TEMP_DEB="$(mktemp)" \
    && wget -O "$TEMP_DEB" 'http://storage.googleapis.com/tensorflow-serving-apt/pool/tensorflow-model-server-1.5.0/t/tensorflow-model-server/tensorflow-model-server_1.5.0_all.deb' \
    && dpkg -i "$TEMP_DEB" \
    && rm -f "$TEMP_DEB" 

# Download the trained model for serving
RUN wget SOME_HOSTED_URL/model-export.zip -O model_export.zip \
    && unzip model_export.zip -d model-export \
    && mv model-export/ /tmp \
    && rm model_export.zip

EXPOSE 9000

# Serve the model when the container starts
ENTRYPOINT ["tensorflow_model_server"]
CMD ["--port=9000", "--model_name=model", "--model_base_path=/tmp/model-export"]

@strategywise
Copy link

strategywise commented Mar 26, 2018

@dcyoung Does this update work on GPU enabled models?

@dcyoung
Copy link

dcyoung commented Mar 27, 2018

@strategywise Whenever I create docker container intended for GPU support, I have them build tensorflow serving from source. I haven't tested the package in that scenario.

@chleibig
Copy link

chleibig commented Apr 1, 2018

While the solution suggested by @KillPinguin (thx!) above works fine (on ubuntu 16.04), it suffices to just update libstdc++6, as suggested by the actual error:

add-apt-repository ppa:ubuntu-toolchain-r/test -y
apt-get update
apt-get install libstdc++6

@cchung100m
Copy link

Hi @KillPinguin,

Thank you for the suggestion. I solve the same issue with update/upgrade command.
It's odd that the update/upgrade command didn't change my GLIBC/CXXAB version, upgrade my tensorflow-model-server version from 1.7.0 to 1.8.0 instead. I think maybe use tensorflow-model-server 1.8.0 directly is another workaround to this issue.

apt-get update
apt-get upgrade
apt-get dist-upgrade

System Information:

root@59dbb35082c6:/# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_DEBUG_MESSAGE_LENGTH

root@59dbb35082c6:/notebooks/downloads/installation_file# strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep CXXAB
CXXABI_1.3
CXXABI_1.3.1
CXXABI_1.3.2
CXXABI_1.3.3
CXXABI_1.3.4
CXXABI_1.3.5
CXXABI_1.3.6
CXXABI_1.3.7
CXXABI_1.3.8
CXXABI_1.3.9
CXXABI_TM_1
CXXABI_FLOAT128
root@59dbb35082c6:/# apt show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.8.0
Built-Using: Bazel
Priority: optional
Section: contrib/devel
Maintainer: TensorFlow Serving team
Installed-Size: unknown
Homepage: https://github.com/tensorflow/serving
Download-Size: 98.0 MB
APT-Manual-Installed: yes
APT-Sources: http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages
Description: TensorFlow Serving ModelServer

root@59dbb35082c6:/# apt-cache show tensorflow-model-server
Package: tensorflow-model-server
Version: 1.8.0
Architecture: all
Maintainer: TensorFlow Serving team
Priority: optional
Section: contrib/devel
Filename: pool/tensorflow-model-server/t/tensorflow-model-server/tensorflow-model-server_1.8.0_all.deb
Size: 97987044
SHA256: 3a3288a964402a5c7f1217ff26d3f971adfbd8a2a2206087f0ea61662cca1d6b
SHA1: 6e446cbdd6fe63c661899c2b14ebf8c0ab3a8df9
MD5sum: 79730a9f53b26e231e08e89b5fe664b9
Description: TensorFlow Serving ModelServer
Description-md5: 9b7e03f5296f318009581d6e285e2f89
Homepage: https://github.com/tensorflow/serving
Built-Using: Bazel

@gautamvasudevan
Copy link
Collaborator

The latest docker images have all the instructions needed to build on Ubuntu 16.04. Closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests