Skip to content

Make setup_logging() more Jupyter-friendly. #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 10, 2021

Conversation

zakv
Copy link
Contributor

@zakv zakv commented Jun 7, 2021

It can be helpful to import code from labscript programs into Jupyter notebooks, particularly for development and debugging purposes. For example, I'm currently using the runviewer.__main__.Shot class to generate some plots. However, with the current version of labscript_utils, running from runviewer.__main__ import Shot from a Jupyter notebook fails, so the code cannot be imported. In the past I've seen this issue when importing some other labscript components as well.

The import fails because importing runviewer.__main__ calls labscript_utils.setup_logging.setup_logging(), which in turn raises an UnsupportedOperation error (example traceback below). The issue is that Jupyter kernels do some special things with stdout and stderr, which means that sys.stdout.fileno and sys.stderr.fileno exist but aren't callable in a Jupyter notebook. This PR changes setup_logging() to catch the UnsupportedOperation error thrown when it is run (possibly indirectly) from a Jupyter notebook. When the error is caught, logging output isn't directed to stdout or stderr, but is still sent to the log files. Note that setting sys.stdout = sys.stderr = open(os.devnull, 'w') isn't a good idea in this case because it prevents print() statements (and likely other output as well) from working in the Jupyter notebook.

This might be too niche of a problem to be worth merging. That said, it would be nice to be able to work on labscript code in a jupyter notebook and I have run into this issue multiple times.

Example traceback:

---------------------------------------------------------------------------
UnsupportedOperation                      Traceback (most recent call last)
<ipython-input-1-a4e555dfbf6f> in <module>
----> 1 from runviewer.__main__ import Shot as RVShot

c:\users\username\software\labscript\runviewer\runviewer\__main__.py in <module>
     37 splash.update_text('importing labscript suite modules')
     38 from labscript_utils.setup_logging import setup_logging
---> 39 logger = setup_logging('runviewer')
     40 labscript_utils.excepthook.set_logger(logger)
     41 

c:\users\username\software\labscript\labscript-utils\labscript_utils\setup_logging.py in setup_logging(program_name, log_level, terminal_level, maxBytes, backupCount)
     53     handler.setLevel(log_level)
     54     logger.addHandler(handler)
---> 55     if sys.stdout is not None and sys.stdout.fileno() >= 0:
     56         stdout_handler = logging.StreamHandler(sys.stdout)
     57         stdout_handler.setFormatter(formatter)

UnsupportedOperation: fileno

@zakv
Copy link
Contributor Author

zakv commented Jul 22, 2021

I also came across this issue here: labscript-suite/blacs#88 (comment) (see the note at the end of that comment)

@dihm
Copy link
Contributor

dihm commented Nov 8, 2021

I actually really like this change and think it should be merged. I, too, like to develop code using jupyter notebooks. Being able to do that with labscript code as well would be appreciated. If nothing else, I know of a few labscript devices that ship with their own drivers so to even test the drivers you have to be able to import labscript. I think this alone makes this a fairly necessary addition.

I would suggest that some sort of warning gets emitted somewhere that signifies this is happening. The concern I have is this catches a fairly generic error just to determine the very specific case that code is being run from jupyter. What if that error is emitted for something unrelated and this silently kills logging to the stdout? I think adding a log message saying the shell redirect has been skipped would cover that base and shouldn't make this any more complicated (hopefully, tell me if I'm wrong).

@zakv
Copy link
Contributor Author

zakv commented Nov 10, 2021

Good idea. I added a warning and also changed the try/except block into a try/except/else construct so that an UnsupportedOperation is only caught if it is raised by sys.stdout.fileno(). That should reduce the chance that this generic error is mistakenly caught.

@dihm
Copy link
Contributor

dihm commented Nov 10, 2021

This looks good to me, but the docs build is failing. I suspect this is because this branch diverged from main before the docs builds were updated. Could you rebase this branch onto master just to ensure nothing silly is going on?

zakv added 4 commits November 10, 2021 12:15
…id catching any UnsupportedOperation errors thrown by code other than sys.stdout.fileno().
…logging().

Probably best practice to call it just once rather than calling it first to check if it raises an error then again later to actually use the value it returns.
@zakv zakv force-pushed the setup-logging-jupyter branch from 05f8c08 to c4dd2a8 Compare November 10, 2021 17:17
@zakv
Copy link
Contributor Author

zakv commented Nov 10, 2021

Rebased and now the docs build!

@dihm
Copy link
Contributor

dihm commented Nov 10, 2021

Cool! I'm going to merge.

@dihm dihm merged commit b8726ad into labscript-suite:master Nov 10, 2021
@zakv zakv deleted the setup-logging-jupyter branch November 10, 2021 22:31
dihm added a commit that referenced this pull request Nov 30, 2021
commit a79d0e4
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Tue Nov 30 11:20:27 2021 -0500

    Update setup.cfg in preparation for release.

commit 811b43d
Merge: b8726ad a3be0db
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 17:07:51 2021 -0500

    Merge pull request #81 from chrisjbillington/py36-path-bug

    Fix bug on Python3.6

commit b8726ad
Merge: d309a5f c4dd2a8
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 17:06:47 2021 -0500

    Merge pull request #78 from zakv/setup-logging-jupyter

    Make setup_logging() more Jupyter-friendly.

commit d309a5f
Merge: d5dfea7 12db878
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 17:06:02 2021 -0500

    Merge pull request #82 from dihm/zlog_port_profile

    Add default zlog port to default profile configuration file.

commit c4dd2a8
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 9 20:12:19 2021 -0500

    Removed duplicate call to sys.stdout.fileno() in setup_logging.setup_logging().

    Probably best practice to call it just once rather than calling it first to check if it raises an error then again later to actually use the value it returns.

commit 2e38107
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 9 20:04:37 2021 -0500

    Restructured try/except block in setup_logging.setup_logging() to avoid catching any UnsupportedOperation errors thrown by code other than sys.stdout.fileno().

commit 97e70c7
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 9 19:56:07 2021 -0500

    Added warning message when setup_logging.setup_logging() cannot send output to stdout and stderr.

commit b711923
Author: Zak V <zakven@mit.edu>
Date:   Mon Jun 7 17:48:41 2021 -0400

    setup_logging.py is now more Jupyter-friendly.

commit d5dfea7
Merge: 9b76dcb b149e19
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 08:54:21 2021 -0500

    Merge pull request #74 from zakv/unitconversions-import-bugfix

    Fixed bug in unitconversions._All._import_all()

commit 12db878
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Mon Nov 8 10:20:28 2021 -0500

    Add default zlog port to default profile configuration file.

commit 9b76dcb
Merge: e13c992 e5cf3fb
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Fri Oct 22 17:53:17 2021 -0400

    Merge pull request #80 from dihm/default_labconfig_docs

    Add default labconfig file to docs

commit a3be0db
Author: chrisjbillington <chrisjbillington@gmail.com>
Date:   Tue Sep 21 13:45:13 2021 +1000

    Fix bug on Python3.6, where Path object not able to be treated as a
    string in ConfigParser.

commit e5cf3fb
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Tue Aug 31 15:29:45 2021 -0400

    Create a doc page that displays the current default labconfig.ini file,
    for easy reference.

commit e13c992
Merge: e74472c 461dc9f
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Fri Jul 16 15:12:37 2021 -0400

    Merge pull request #79 from dihm/labscript_utils-docs

    Update docs to match autogenerating templates used in other modules of labscript.

commit 461dc9f
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Thu Jul 15 10:26:41 2021 -0400

    Add docstring coverage test to build.

commit f90a4cc
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Thu Jul 15 10:26:26 2021 -0400

    Update sphinx version.

commit e45107d
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Mon Jul 12 12:57:18 2021 -0400

    Converting docs build to fully automated version based on recursive
    autosummary calls.

    This more closely resembles how docs are built in the other modules,
    but is slightly different in that submodules are also recursively generated.

commit 56d64f7
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Mon Jul 12 09:25:18 2021 -0400

    Update sphinx pins to same versions as rest of suite.

commit e74472c
Merge: f0999f5 c543328
Author: Chris Billington <chrisjbillington@gmail.com>
Date:   Tue Feb 9 08:54:16 2021 +1100

    Merge pull request #76 from zakv/fix-27

    Work around infinite recursion in ToolPalette.resizeEvent()

commit c543328
Author: Zak V <zakven@mit.edu>
Date:   Mon Feb 8 13:30:47 2021 -0500

    Removed some python 2 debugging print statements from toolpalette.py.

commit d9eb80f
Author: Zak V <zakven@mit.edu>
Date:   Sun Feb 7 23:44:47 2021 -0500

    Fixed #27; worked around bug where ToolPalette.resizeEvent() could trigger itself in an infinite resursion loop.

commit f0999f5
Merge: 8371585 7d14b65
Author: Chris Billington <chrisjbillington@gmail.com>
Date:   Mon Jan 4 13:46:54 2021 +1100

    Merge pull request #75 from chrisjbillington/master

    Do not use deprecated set-env command

commit 7d14b65
Author: chrisjbillington <chrisjbillington@gmail.com>
Date:   Mon Jan 4 13:40:14 2021 +1100

    Do not use deprecated set-env command

commit b149e19
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 24 18:06:11 2020 -0500

    Fixed bug in unitconversions._All._import_all() where self.__all__ wasn't always changed from None to an empty list before appending things to it.
dihm added a commit that referenced this pull request Mar 21, 2022
commit 95965b8
Merge: a88582d f2f0ac7
Author: Chris Billington <chrisjbillington@gmail.com>
Date:   Thu Mar 17 08:02:09 2022 +1100

    Merge pull request #86 from dihm/replace_distutils

    Replace distutils

commit f2f0ac7
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Wed Mar 16 09:43:51 2022 -0400

    Replace `distutils.log` with basic `logging` in `setup.py`.

commit b6b7823
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Wed Mar 16 09:04:46 2022 -0400

    Replace `distutils.sysconfig` with stdlib `sysconfig` in `modulewatcher`.

commit 9840228
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Wed Mar 16 09:04:10 2022 -0400

    Replace `distutils.version.LooseVersion` with `packaging.version.Version` in `ls_zprocess`.

commit a88582d
Merge: 66c68ab 75a7df6
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Thu Mar 3 09:28:57 2022 -0500

    Merge pull request #84 from dihm/generic_freq_conv

    Add generic frequency conversion class

commit 75a7df6
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Feb 9 12:27:17 2022 -0500

    Add generic frequency conversion class that goes between standard SI prefixes from a base of Hz.

commit 66c68ab
Merge: 7a3d58c 27501a3
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Feb 9 10:47:36 2022 -0500

    Merge pull request #83 from dihm/fix_docs_build

    Prevent starting a zlock server when importing `h5_lock` on RTD.

commit 27501a3
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Feb 9 10:34:22 2022 -0500

    Prevent starting a zlock server when importing `h5_lock` on RTD.

commit 7a3d58c
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Feb 2 17:24:26 2022 -0500

    Pin `mistune` to fix doc builds on RTD.

commit 36c3c6a
Author: chrisjbillington <chrisjbillington@gmail.com>
Date:   Fri Jan 7 17:23:48 2022 +1100

    Ensure only ints passed to QFrame.move in splash

    Otherwise this is a TypeError in the latest PyQt5.

commit f4da12e
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Tue Nov 30 13:00:27 2021 -0500

    Triggering build of release branch by undoing unnecessary pin change for pyqtgraph.

commit a79d0e4
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Tue Nov 30 11:20:27 2021 -0500

    Update setup.cfg in preparation for release.

commit 811b43d
Merge: b8726ad a3be0db
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 17:07:51 2021 -0500

    Merge pull request #81 from chrisjbillington/py36-path-bug

    Fix bug on Python3.6

commit b8726ad
Merge: d309a5f c4dd2a8
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 17:06:47 2021 -0500

    Merge pull request #78 from zakv/setup-logging-jupyter

    Make setup_logging() more Jupyter-friendly.

commit d309a5f
Merge: d5dfea7 12db878
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 17:06:02 2021 -0500

    Merge pull request #82 from dihm/zlog_port_profile

    Add default zlog port to default profile configuration file.

commit c4dd2a8
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 9 20:12:19 2021 -0500

    Removed duplicate call to sys.stdout.fileno() in setup_logging.setup_logging().

    Probably best practice to call it just once rather than calling it first to check if it raises an error then again later to actually use the value it returns.

commit 2e38107
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 9 20:04:37 2021 -0500

    Restructured try/except block in setup_logging.setup_logging() to avoid catching any UnsupportedOperation errors thrown by code other than sys.stdout.fileno().

commit 97e70c7
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 9 19:56:07 2021 -0500

    Added warning message when setup_logging.setup_logging() cannot send output to stdout and stderr.

commit b711923
Author: Zak V <zakven@mit.edu>
Date:   Mon Jun 7 17:48:41 2021 -0400

    setup_logging.py is now more Jupyter-friendly.

commit d5dfea7
Merge: 9b76dcb b149e19
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Wed Nov 10 08:54:21 2021 -0500

    Merge pull request #74 from zakv/unitconversions-import-bugfix

    Fixed bug in unitconversions._All._import_all()

commit 12db878
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Mon Nov 8 10:20:28 2021 -0500

    Add default zlog port to default profile configuration file.

commit 9b76dcb
Merge: e13c992 e5cf3fb
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Fri Oct 22 17:53:17 2021 -0400

    Merge pull request #80 from dihm/default_labconfig_docs

    Add default labconfig file to docs

commit a3be0db
Author: chrisjbillington <chrisjbillington@gmail.com>
Date:   Tue Sep 21 13:45:13 2021 +1000

    Fix bug on Python3.6, where Path object not able to be treated as a
    string in ConfigParser.

commit e5cf3fb
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Tue Aug 31 15:29:45 2021 -0400

    Create a doc page that displays the current default labconfig.ini file,
    for easy reference.

commit e13c992
Merge: e74472c 461dc9f
Author: David Meyer <dihm@users.noreply.github.com>
Date:   Fri Jul 16 15:12:37 2021 -0400

    Merge pull request #79 from dihm/labscript_utils-docs

    Update docs to match autogenerating templates used in other modules of labscript.

commit 461dc9f
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Thu Jul 15 10:26:41 2021 -0400

    Add docstring coverage test to build.

commit f90a4cc
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Thu Jul 15 10:26:26 2021 -0400

    Update sphinx version.

commit e45107d
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Mon Jul 12 12:57:18 2021 -0400

    Converting docs build to fully automated version based on recursive
    autosummary calls.

    This more closely resembles how docs are built in the other modules,
    but is slightly different in that submodules are also recursively generated.

commit 56d64f7
Author: David Meyer <dihm.meyer@gmail.com>
Date:   Mon Jul 12 09:25:18 2021 -0400

    Update sphinx pins to same versions as rest of suite.

commit e74472c
Merge: f0999f5 c543328
Author: Chris Billington <chrisjbillington@gmail.com>
Date:   Tue Feb 9 08:54:16 2021 +1100

    Merge pull request #76 from zakv/fix-27

    Work around infinite recursion in ToolPalette.resizeEvent()

commit c543328
Author: Zak V <zakven@mit.edu>
Date:   Mon Feb 8 13:30:47 2021 -0500

    Removed some python 2 debugging print statements from toolpalette.py.

commit d9eb80f
Author: Zak V <zakven@mit.edu>
Date:   Sun Feb 7 23:44:47 2021 -0500

    Fixed #27; worked around bug where ToolPalette.resizeEvent() could trigger itself in an infinite resursion loop.

commit f0999f5
Merge: 8371585 7d14b65
Author: Chris Billington <chrisjbillington@gmail.com>
Date:   Mon Jan 4 13:46:54 2021 +1100

    Merge pull request #75 from chrisjbillington/master

    Do not use deprecated set-env command

commit 7d14b65
Author: chrisjbillington <chrisjbillington@gmail.com>
Date:   Mon Jan 4 13:40:14 2021 +1100

    Do not use deprecated set-env command

commit b149e19
Author: Zak V <zakven@mit.edu>
Date:   Tue Nov 24 18:06:11 2020 -0500

    Fixed bug in unitconversions._All._import_all() where self.__all__ wasn't always changed from None to an empty list before appending things to it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants