Unable to use tensorboard viewer using minio path

[ramdootp]

the tensorboard pod is running with the above metadata.json
But it fails, reporting minio style path isn't supported by tensorboard.

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 270, in run_wsgi
execute(self.server.app)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 258, in execute
application_iter = app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/tensorboard/backend/application.py", line 307, in call
return self.data_applications[clean_path](environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/wrappers.py", line 308, in application
resp = f(*args[:-2] + (request,))
File "/usr/local/lib/python2.7/dist-packages/tensorboard/backend/application.py", line 278, in _serve_plugins_listing
response[plugin.plugin_name] = plugin.is_active()
File "/usr/local/lib/python2.7/dist-packages/tensorboard/plugins/beholder/beholder_plugin.py", line 71, in is_active
return tf.gfile.Exists(summary_filename) and
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 260, in file_exists
pywrap_tensorflow.FileExists(compat.as_bytes(filename), status)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
UnimplementedError: File system scheme 'minio' not implemented (file: 'minio://tensorboard/plugins/beholder/frame.summary')

Is this the correct way to use the minio path for metadata.json?
if not, let me know the correct way.

[yebrahim]

Tensorboard does not support reading from minio server, we're working on a way to use mounted volumes to save this data, in which case Tensorboard will just be reading from the local filesystem. See the discussion in #489.

cc @Ark-kun

[ramdootp]

Thanks @yebrahim

Is it the same with confusion matrix viewer, we are seeing similar issue. The display shows blank screen

[paveldournov]

@yebrahim let's use this as a tracking bug for the TB running off mounted volumes.

/assign @yebrahim

[vicaire]

/cc @neuromage

It seems that we could add support for mounted volumes in the viewer CRD for tensorboard that @neuromage did

[swiftdiaries]

I'd like to take this up and look into using the Viewer CRD from the Frontend for Tensorboard following discussions here: #792 (comment)
/cc @yebrahim
/assign @swiftdiaries

[vicaire]

Thank you @swiftdiaries

[rileyjbauer]

@neuromage can this be closed with the Viewer CRD work Jing has done?

[neuromage]

I think this is technically possible (mounting volumes and having the TB pod read off that), but we don't provide the user with the ability to do this from our UI as yet. @swiftdiaries are you still interested in working on this?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[Dhaval08]

Does Kubeflow Pipeline now support local source for visualization?

[Bobgy]

It supports inline source, but not minio

[Bobgy]

/reopen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Junwen-xie]

It supports inline source, but not minio

hi, we are testing tensorboard in pipline, the documentation said not support inline source....

however we test souce in mino, it dose not work as well..

[Bobgy]

Ohh, sorry I missed you are talking about tensorboard. Inline is not supported

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kd303]

Hi @Bobgy , @swiftdiaries

Is it possible to use a PV for Tensorboard working on non-cloud environments (local cluster running with Kubeflow 1.1) ? Also if possible how can I make it work? Like if I crate a PV and mount the PV on training container and whether same PV can be mounted on Tensorboard container - however I am not aware of Tensorboard container definitions or even how they spins up in Kubeflow pipelines.. any insights would be helpful.

[blacksailer]

@kd303 Hi

I've managed to use on local cluster with adding pre-defined PV. Here, in this issue you can find sample configs to tensorboard component.

But for training component I have used PodDefault to attach the same volume to master/worker pods(I am using PyTorchJob component).

In result, in pipelines UI I can see two components: Tensorboard and PyTorch Component. (I have edited this TfJob component to create PyTorch one)

[kd303]

@blacksailer Thanks, I will give it a go,

[Bobgy]

Hi all, I'm sending a PR to make it possible to use tensorboard viewer with minio path: #5515

Feedback welcomed!