TensorBoard Integration with Notebook Example #653

closedLoop · 2017-05-27T19:52:14Z

Integration of TensorBoard with Notebook Example

@dustinvtran here's a quick demo of the changes
https://www.useloom.com/share/29e087b17cfc47b9b60662bff7843983

Contents:

Modification of inference/inference.py to more completely initialize Tensorboard
A Notebook with instructions and best practices to make your TensorBoard clean and useful

Main Features:

Automatically tracks all named latent variables for your given inference
- This can be disabled or modified by passing in a custom list via logvars
Works for all inference classes.
Automatic run versioning to keep your logs clean and comparable in TensorBoard
Instructions for people to easily configure and launch their model with TensorBoard

Issues addressed

Edward Roadmap #464 satisfies part of the 'More TensorBoard support'
tf.name_scope()'s and tensorboard #190 the Notebook shows best practices for TensorBoard to look clean using Edward
visualization of specified models #351 Partially addresses this by automatically visualizing all named latent variables
visualization of inference/convergence diagnostics #353 This shows parameter MCMC values, but not loss functions.

Not addressed:

Storing inference specific loss and convergence information. I saw that other were working on this and so I didn't touch it. tensorboard for variational inference #598

…ibutions

Vis tensorboard

…cyteResearch/edward into example_multivariatenormal

dustinvtran · 2017-05-28T00:56:03Z

edward/inferences/inference.py

    debug : bool, optional
      If True, add checks for ``NaN`` and ``Inf`` to all computations
      in the graph. May result in substantially slower execution
      times.
    """
+    from datetime import datetime
+    import os


Can you move import statements to the top of the file?

dustinvtran · 2017-05-28T01:00:24Z

edward/inferences/inference.py

@@ -169,11 +171,25 @@ def initialize(self, n_iter=1000, n_print=None, scale=None, logdir=None,
    logdir : str, optional
      Directory where event file will be written. For details,
      see ``tf.summary.FileWriter``. Default is to write nothing.
+    logrun : str, optional


Could you describe your reasoning for splitting logdir vs logrun? I think I see it, although I wonder if logdir could just be made more powerful by letting you include a custom subdirectory naming. If unspecified, it would default to a subdirecty with the current UTC timestamp.

Also: if logrun can easily be specified as part of logdir that would also enable the timestampping for those who prefer to control inference using initialize() then updates(). (that is, they don't call run())

They way I thought about it is that logrun will tend to be pretty static for a given project and you want a way to cleanly separate info each time you refit your models. Therefor you can leave it blank as you arbitrarily refine the model and use the timestamps to track your progress. Else you could set logrun to something like using_adam vs using_adagrad to highlight in TensorBoard the convergence results of each.

I wanted to keep logdir a string since it is a string in TensorBoard, and given it has the same name I wanted to make the usage similar to avoid confusion.

Users can already separate info each time they refit a model by tweaking the logdir argument. A typical use case would be

inference.initialize(logdir='log/lr_{}_nsamples_{}'.format(learning_rate_stepsize, n_samples))

This is how native TensorFlowers write the logdir argument in their tf.summary.FileWriter. Users have full flexibility in nesting however many subdirectories and filenames they want. logdir + logrun sort of imposes redundancy, where a user can specify logrun already inside logdir.

That said, I love the idea of having a default subdirectory appended to logdir which timestamps the run. It seems impossible to turn it on/off without having an additional argument. So then maybe logrun should just be a boolean, and its name+default be changed to something like log_timestamp=True?

dustinvtran · 2017-05-28T01:02:43Z

edward/inferences/inference.py

+
+      if logrun is None:
+        # Set default to timestamp
+        logrun = datetime.strftime(datetime.utcnow(), "%Y%m%dT%H%M%S")


Good standard! What's the T?

Also, do you think adding _ inbetween would make the directory name too long? (Probably.) Is there a standard other code bases have done regarding this which we might follow?

The T was a means of just separating the date and the time segments like most of the conventions here https://www.loggly.com/docs/automated-parsing/#system and also removing all of the other non-numeric characters from the conventions. However after using this in tensorboard a bit. I'm going to replace the 'T' with a '_' just to keep it easier to read on the eye.

dustinvtran · 2017-05-28T01:03:00Z

edward/inferences/inference.py

@@ -258,3 +284,48 @@ def finalize(self):
    """
    if self.logging:
      self.train_writer.close()
+
+  def set_log_variables(self, logvars=None, log_max_scalers_per_var=None):
+    """Logs variables to TensorBoard


Tensorboard.

dustinvtran · 2017-05-28T01:04:52Z

edward/inferences/inference.py

+    """Logs variables to TensorBoard
+
+     For each variable in logvars, creates 'scalar' and / or 'histogram' by
+     calling `tf.summary.scalar` or `tf.summary.histogram`


To format code, wrap double single ticks, i.e., \``tf.summary.scalar``\ (without the \). Also needs a period at end of sentence.

dustinvtran · 2017-05-28T01:08:03Z

edward/inferences/inference.py

+    if logvars is None:
+      logvars = []
+      for k in self.latent_vars:
+        logvars += get_variables(self.latent_vars[k])


variables can also exist as model parameters. I recommend an additional loop applying this to keys of self.data and the priors (which are keys of self.latent_vars).

dustinvtran · 2017-05-28T01:13:31Z

edward/version.py

@@ -1 +1 @@
-__version__ = '1.3.1'
+__version__ = '1.3.1-closedloop'


Can remove this :)

dustinvtran · 2017-05-28T01:14:09Z

examples/normal_sghmc.py

@@ -50,6 +50,7 @@ def mvn_plot_contours(z, label=False, ax=None):
 qz = Empirical(params=tf.Variable(tf.random_normal([5000, 2])))

 inference = ed.SGHMC({z: qz})
+inference.initialize(logdir='C:\\Users\skruz\\temp\\tensorflow\\')


You can remove this argument to be more generic, say, logdir='log'.

dustinvtran · 2017-05-28T01:18:09Z

This is a great PR @closedLoop. It's a valuable contribution in making Edward fits easier to explore and also to diagnose model/learning problems. And thanks for the demo!

I added comments above. Two others

Can you remove changes from the various notebooks unrelated to this PR?
I recommend displaying the TensorBoard pics in the tutorial. You can do that by using something like

![GAN Fig 0](https://raw.githubusercontent.com/blei-lab/edward/master/docs/images/gan-fig0.png)

Then you can explain them as you do in the video demo.

dustinvtran · 2017-05-28T08:19:42Z

edward/inferences/inference.py

+
+      # If var is multi-dimensional, log the distribution
+      if len(var.shape) > 0 and np.max(var.shape) > 1:
+        tf.summary.histogram(var_name, var)


Some of this conflicts with the current tensorboard tracking in variational_inference.py (see #654). My guess is you can remove the parameter histograms in variational_inference.py.

I also recommend checking to see how the naming systems match.

…quest

Reverting multivariate normal changes not related to TensorBoard

update with latest master

get up-to-date

Get update code from master

…h/edward into vis_tensorboard

closedLoop · 2017-05-29T20:46:19Z

@dustinvtran this has been cleaned up integrated with the gradient and loss tracking from the merged pull request #598 . I think it's looking really good and I've tested it on some larger models and it works well there too.

Main changes:

parameter naming is consistent with logging in variational_inference.py
logrun has been changed to log_timestamp boolean based on your suggestion.
I've removed log_max_scalars_per_var from my previous addition. It was unnecessarily complicated and was a solution in search of a problem.
Single valued Variables are graphed in Scalars else they are logged to Histogram and Distributions in TensorBoard.
The tensorboard.ipynb has been updated and includes references to the gradient tracking and loss logged that was added in tensorboard for variational inference #598

Thank you for all of your suggestions. I believe this is much cleaner now. I'm happy to modify anything else you'd like

Here's the new screenshots!

dustinvtran · 2017-05-30T03:07:51Z

Great work. Merging now. I'll add a separate PR that adds the equivalent notebook as a tutorial on the website.

= and others added 12 commits May 27, 2017 10:59

WIP multivariate normal betas estimated propery

5e9f765

WIP

2e62c5f

added logrun parameter to initialize

419a722

Added tensorboard example notebook

36f6680

added default logging options for latent variables, scalars and distr…

80d27b4

…ibutions

Added screen tensorboard screenshots

8f68239

Final minor typos

1585334

pep8 fixes

fadd7f1

whitespace

50900c2

Merge branch 'example_multivariatenormal' into vis_tensorboard

3e81dfc

Merge pull request #1 from AstrocyteResearch/vis_tensorboard

f0ab25d

Vis tensorboard

Merge branch 'example_multivariatenormal' of https://github.com/Astro…

5a5ae36

…cyteResearch/edward into example_multivariatenormal

dustinvtran reviewed May 28, 2017

View reviewed changes

dustinvtran mentioned this pull request May 28, 2017

Prettify TensorBoard scalars and histogram summaries #654

Merged

= and others added 12 commits May 28, 2017 10:05

reverting changes in documents to related to this TensorBoard pull re…

fc99145

…quest

incorporated @dustinvtran feedback

838c485

Merge pull request #2 from AstrocyteResearch/example_multivariatenormal

637ffaa

Reverting multivariate normal changes not related to TensorBoard

pep8 compliance

7a36a53

Merge pull request #3 from AstrocyteResearch/master

675912a

update with latest master

Merge pull request #4 from blei-lab/master

55d52a2

get up-to-date

Merge pull request #5 from AstrocyteResearch/master

34cfb35

Get update code from master

changed logrun to log_timestamp and logvars to log_vars

2d9ac58

updated example notebook with better descriptions of screenshots

7a2410b

Merge branch 'vis_tensorboard' of https://github.com/AstrocyteResearc…

5667f02

…h/edward into vis_tensorboard

aligned logging naming conventions and simplified logging configuration

e1e27e2

updated tutorial and images to include gradients and parameter graphs

29eca71

Update supervised_regression.ipynb

b4723b1

Update iclr2017.ipynb

a22b546

dustinvtran merged commit a17d650 into blei-lab:master May 30, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TensorBoard Integration with Notebook Example #653

TensorBoard Integration with Notebook Example #653

closedLoop commented May 27, 2017

dustinvtran May 28, 2017

dustinvtran May 28, 2017

dustinvtran May 28, 2017

closedLoop May 28, 2017

dustinvtran May 28, 2017 •

edited

Loading

dustinvtran May 28, 2017 •

edited

Loading

closedLoop May 28, 2017

dustinvtran May 28, 2017

dustinvtran May 28, 2017

dustinvtran May 28, 2017

dustinvtran May 28, 2017

dustinvtran May 28, 2017

dustinvtran commented May 28, 2017 •

edited

Loading

dustinvtran May 28, 2017

closedLoop commented May 29, 2017

dustinvtran commented May 30, 2017

		@@ -1 +1 @@
		__version__ = '1.3.1'
		__version__ = '1.3.1-closedloop'

TensorBoard Integration with Notebook Example #653

TensorBoard Integration with Notebook Example #653

Conversation

closedLoop commented May 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustinvtran May 28, 2017 • edited Loading

Choose a reason for hiding this comment

dustinvtran May 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustinvtran commented May 28, 2017 • edited Loading

Choose a reason for hiding this comment

closedLoop commented May 29, 2017

dustinvtran commented May 30, 2017

dustinvtran May 28, 2017 •

edited

Loading

dustinvtran May 28, 2017 •

edited

Loading

dustinvtran commented May 28, 2017 •

edited

Loading