Show trial logs on the interface

[prcastro]

/kind feature

Describe the solution you'd like

The problem is accessing and debugging the trials of a model. A possible solution is to have, on the interface, a way to access the logs of each trial. Seeing what trials are running and which ones are failing are a plus.

[johnugeorge]

/area frontend

/cc @andreyvelich

[k8s-ci-robot]

@johnugeorge: The label(s) area/frontend cannot be applied, because the repository doesn't have them

In response to this:

/area frontend

/cc @andreyvelich

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[johnugeorge]

/area front-end

[nlarusstone]

This is also a question we have been having -- how are we intended to access the trial logs currently?

[issue-label-bot]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[andreyvelich]

@nlarusstone Right now, you can access Trial logs using kubectl or K8s APIs with corresponding training container.
In the future releases we have plans to add this feature on the UI as well.
/priority p2

[nlarusstone]

Forgive my ignorance, but where is the training container actually run? I can't seem to run kubectl logs <trial-name>

[andreyvelich]

@nlarusstone No worries!
If you want to save Trial's pods after completion, you have to specify retain: true in Experiment, e.g. like here.

After that, you should be able to check logs from the created pods even if Trial is complete.
Trial's job should run in the same namespace as your Experiment.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen