[rllib] Feature/histograms in tensorboard

[roireshef]

Why are these changes needed?

Histograms are super powerful tool to use when the relevant information is hard to get compressed into a single scalar (in case one would be interested in the full distribution, for example).

Related issue number

Closes #4953

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.

[AmplabJenkins]

Can one of the admins verify this patch?

[thavlik]

This is pretty exciting. We really need to get epsiode_reward_[min/max/mean] visualized with TB histograms.

For clarification (and so everyone else doesn't have to look at the code) histogram data is located under info['episode']['hist_data'], and we specify dict with each member being a float array? Example:

info['episode']['hist_data'] = {'my_histogram_data':[1.0, 0.5, 0.23, 0.4]}

Are deeply nested hist_data members supported? Does info['episode']['hist_data'] = {'my_nested':{'my_histogram_data':{[1.0, 0.5, 0.23, 0.4]}} show up as my_nested/my_histogram_data in TB?

Thank you!!

EDIT: It could also be useful to report training stats from tune.run with num_samples > 1 as TB histograms. Suppose you had a large number of trials with matching hparams, it would cleanly represent how the training dynamics change for the population over time.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21115/
Test FAILed.

[roireshef]

@thavlik no nesting here. info['episode'].hist_data[<metric_name>] expects a list of raw values, from which it will compute binned frequencies. See the example in the second commit. In the current implementation, I guess you could name the metric "my_nested/my_histogram_data" to get what you desire, but I haven't tried that.

[richardliaw]

Thanks for this change! cc @ericl take a look?

[ericl]

@richardliaw we are using TBXLogger by default now right? So in Ray latest to_tf_values won't be called.

I think this logic should be moved to the TBXLogger class.

[ericl]

(we should also delete this old code).

[richardliaw]

@roireshef can you push this logic into the TBXLogger class?

[roireshef]

I think I got it right, but can't test because I can't get this version to run inside my docker container yet (see #6662). It might take me a while to figure out, so I'd be glad to get your help testing it if you could...

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21166/
Test PASSed.

[ericl]

This looks great! Just tried it out (also pushed a commit to record histograms for the episode stats by default):

[ericl]

Looks good pending test run.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/21176/
Test FAILed.

[roireshef]

Glad you like it! Be sure to checkout the Histogram tab as well. I found it more interpretable than the Distribution one