Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dashboard missing when starting in a Docker container #1875

Open
jakirkham opened this issue Mar 30, 2018 · 21 comments
Open

Dashboard missing when starting in a Docker container #1875

jakirkham opened this issue Mar 30, 2018 · 21 comments
Labels
documentation Improve or add to documentation good first issue Clearly described and easy to accomplish. Good for beginners to the project.

Comments

@jakirkham
Copy link
Member

jakirkham commented Mar 30, 2018 •
edited
Loading

For some reason, starting distributed in a Docker container results in the bokeh-based dashboard being unavailable. This is when the correct port (8787) is forwarded from the container. Have also verified that the Bokeh dashboard is running on that port in the container. Strangely the same issue does not occur when using dask-drmaa or when using distributed outside of a Docker container.

Example: 
>>> import distributed
>>> cluster = distributed.LocalCluster()
>>> client = distributed.Client(cluster)

Environment: 
name: test
channels:
- conda-forge
- defaults
dependencies:
- blas=1.1=openblas
- bokeh=0.12.14=py36_1
- ca-certificates=2018.1.18=0
- certifi=2018.1.18=py36_0
- click=6.7=py_1
- cloudpickle=0.5.2=py_0
- cytoolz=0.9.0.1=py36_0
- dask-core=0.17.2=py_0
- distributed=1.21.4=py36_0
- heapdict=1.0.0=py36_0
- jinja2=2.10=py36_0
- markupsafe=1.0=py36_0
- msgpack-python=0.5.6=py36_0
- ncurses=5.9=10
- numpy=1.14.2=py36_blas_openblas_200
- openblas=0.2.20=7
- openssl=1.0.2n=0
- packaging=17.1=py_0
- pip=9.0.1=py36_1
- psutil=5.4.3=py36_0
- pyparsing=2.2.0=py36_0
- python=3.6.5=0
- python-dateutil=2.7.2=py_0
- pyyaml=3.12=py36_1
- readline=7.0=0
- setuptools=39.0.1=py36_0
- six=1.11.0=py36_1
- sortedcontainers=1.5.9=py36_0
- sqlite=3.20.1=2
- tblib=1.3.2=py36_0
- tk=8.6.7=0
- toolz=0.9.0=py_0
- tornado=5.0.1=py36_1
- wheel=0.30.0=py36_2
- xz=5.2.3=0
- yaml=0.1.7=0
- zict=0.1.3=py_0
- zlib=1.2.11=0
- libgfortran=3.0.0=1

Using macOS 10.11 with Docker for Mac 18.03.0-ce-mac58 (23607) from the Edge channel. Also used the condaforge/linux-anvil image as a base image with digest sha256:e4dbddbf5c1d1e5143b003608c2a29e6437cdbdb6c4b748cd09fee35f63ab8b3. Though expect any minimal Docker image with a copy of Miniconda to do the install would also work.
@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

Page doesn't load.

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

Right 404.

Will look and report back.

Accessing it on localhost. Would hope that is allowed. :)

Are you able to reproduce it?

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

No worries.

Ran client.get_scheduler_logs() and it looks ok. Confirms Bokeh should be up and running on 8787. Are there more detailed logs I can dive through? Is there a way to get logs from Bokeh?

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

So Docker only shows what gets printed to stdout/stderr, which in the MRE shows nothing. If I run using the same startup mechanism, but with a few more bells and whistles (i.e. Jupyter Notebook) then the logging is just from the Jupyter Notebook.

Is there some way to get access to the Bokeh Client running the dashboard from the Distributed Client somehow?

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

jakirkham commented Mar 30, 2018 •
edited
Loading

Just updating what I have found thus far.

There was a new release of bokeh, 0.12.15, which came out yesterday. Upgrading did not solve the issue.

Inspecting the Bokeh server inside the container, it looks ok. Also, using Requests, am able to load the page inside the container, but not outside the container. Suggesting it is a networking related issue.

If it weren't for the fact that the Jupyter Notebook loads fine outside of the container and dask-drmaa avoids this issue, would blame Docker for it, but that doesn't look like the case.

Edit: Also played with forwarding the Bokeh port to different ports outside the container and had no luck with that either.

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

jakirkham commented Mar 30, 2018 •
edited
Loading

Only seeing a Bokeh warning saying that connections from all hosts are permitted. Given the issue encountered, this is a good thing to have affirmed, but doesn't really explain why it isn't working unfortunately.

@mrocklin
Copy link
Member

mrocklin commented Mar 30, 2018 via email

@jakirkham
Copy link
Member Author

jakirkham commented Mar 30, 2018 •
edited
Loading

Good idea. Ran python -m http.server 8787 and that loaded fine outside the container. Showed a file listing of the directory the server started in within the container.

Edit: Should add this was run when Distributed and the Bokeh server were not running.

@jakirkham
Copy link
Member Author

Also tried serving up this simple Bokeh example using bokeh serve --port 8787 --show myapp.py and that worked fine as well.

@dhirschfeld
Copy link
Contributor

Is the bokeh dashboard itself trying to communicate (pull data) from outside the container?

@jakirkham
Copy link
Member Author

After PR ( #1934 ), it avoids downloading external resources (e.g. logos and the like). Was this the kind of data that you had in mind or were you thinking of something else?

@dhirschfeld
Copy link
Contributor

That and possibly communicating directly with workers. Just grasping at straws really :|

I'm actually going to be spinning up distributed in containers later this week so I'll at least be able to provide independent verification.

@jakirkham
Copy link
Member Author

Would be eager to hear what you find. :)

@jakirkham
Copy link
Member Author

Alright, have narrowed it down. Here is the difference.

Fail:

import distributed
c = distributed.LocalCluster(ip=None)  # default `ip` argument currently

Success:

import distributed
c = distributed.LocalCluster(ip="")

As the ip outside of the container doesn't match the one inside the container, this makes sense.

Not sure that we should change the behavior, but maybe documenting this particular case would be worthwhile (assuming there are not already docs I missed on this :).

@jgerardsimcock jgerardsimcock mentioned this issue Jul 20, 2018
Closed
@GenevieveBuckley GenevieveBuckley added diagnostics documentation Improve or add to documentation good first issue Clearly described and easy to accomplish. Good for beginners to the project. and removed diagnostics labels Oct 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation Improve or add to documentation good first issue Clearly described and easy to accomplish. Good for beginners to the project.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mrocklin @dhirschfeld @jakirkham @GenevieveBuckley