Rationalize logging helpers and docs

shade and openstacksdk each have a logging doc and a logging setup helper
function. They both basically do the same thing, and we have a TODO item
about collapsing them.

This moves openstack.utils.enable_logging to openstack.enable_logging
(leaving behind a compat piece at openstack.utils.enable_logging) It
adds the shade functionality to it, and also changes the behavior to
match shade's WRT behavior when no parameters are passed (defaults to
logging to stdout)

Update the codebase to call openstack._log.setup_logging instead of
logging.getLogger directly, as setup_logging attaches a NullHandler by
default.

Collapse the docs into a single document.

There were only two places where openstacksdk was already logging to
something other than 'openstack'. Collapse those down to 'openstack'
until we come up with a reason to break them out more logically.

Change-Id: I45fd5ffd18255450d38a1f56c80f5c157ea19ae3

README.rst

@@ -6,45 +6,95 @@ with OpenStack clouds. The project aims to provide a consistent and
 complete set of interactions with OpenStack's many services, along with
 complete documentation, examples, and tools.
 
-It also contains a simple interface layer. Clouds can do many things, but
+It also contains an abstraction interface layer. Clouds can do many things, but
 there are probably only about 10 of them that most people care about with any
 regularity. If you want to do complicated things, the per-service oriented
-portions of the SDK are for you. However, if what you want is to be able to
+portions of the SDK are for you. However, if what you want to be able to
 write an application that talks to clouds no matter what crazy choices the
 deployer has made in an attempt to be more hipster than their self-entitled
-narcissist peers, then the ``openstack.cloud`` layer is for you.
+narcissist peers, then the Cloud Abstraction layer is for you.
 
 A Brief History
 ---------------
 
+.. TODO(shade) This history section should move to the docs. We can put a
+   link to the published URL here in the README, but it's too long.
+
 openstacksdk started its life as three different libraries: shade,
 os-client-config and python-openstacksdk.
 
-``shade`` started its life as some code inside of OpenStack Infra's nodepool
-project, and as some code inside of Ansible. Ansible had a bunch of different
-OpenStack related modules, and there was a ton of duplicated code. Eventually,
-between refactoring that duplication into an internal library, and adding logic
-and features that the OpenStack Infra team had developed to run client
-applications at scale, it turned out that we'd written nine-tenths of what we'd
-need to have a standalone library.
+``shade`` started its life as some code inside of OpenStack Infra's `nodepool`_
+project, and as some code inside of the `Ansible OpenStack Modules`_.
+Ansible had a bunch of different OpenStack related modules, and there was a
+ton of duplicated code. Eventually, between refactoring that duplication into
+an internal library, and adding the logic and features that the OpenStack Infra
+team had developed to run client applications at scale, it turned out that we'd
+written nine-tenths of what we'd need to have a standalone library.
+
+Because of its background from nodepool, shade contained abstractions to
+work around deployment differences and is resource oriented rather than service
+oriented. This allows a user to think about Security Groups without having to
+know whether Security Groups are provided by Nova or Neutron on a given cloud.
+On the other hand, as an interface that provides an abstraction, it deviates
+from the published OpenStack REST API and adds its own opinions, which may not
+get in the way of more advanced users with specific needs.
 
 ``os-client-config`` was a library for collecting client configuration for
-using an OpenStack cloud in a consistent and comprehensive manner.
-In parallel, the python-openstacksdk team was working on a library to expose
-the OpenStack APIs to developers in a consistent and predictable manner. After
-a while it became clear that there was value in both a high-level layer that
-contains business logic, a lower-level SDK that exposes services and their
-resources as Python objects, and also to be able to make direct REST calls
-when needed with a properly configured Session or Adapter from python-requests.
-This led to the merger of the three projects.
-
-The contents of the shade library have been moved into ``openstack.cloud``
-and os-client-config has been moved in to ``openstack.config``. The next
-release of shade will be a thin compatibility layer that subclasses the objects
-from ``openstack.cloud`` and provides different argument defaults where needed
-for compat. Similarly the next release of os-client-config will be a compat
+using an OpenStack cloud in a consistent and comprehensive manner, which
+introduced the ``clouds.yaml`` file for expressing named cloud configurations.
+
+``python-openstacksdk`` was a library that exposed the OpenStack APIs to
+developers in a consistent and predictable manner.
+
+After a while it became clear that there was value in both the high-level
+layer that contains additional business logic and the lower-level SDK that
+exposes services and their resources faithfully and consistently as Python
+objects.
+
+Even with both of those layers, it is still beneficial at times to be able to
+make direct REST calls and to do so with the same properly configured
+`Session`_ from `python-requests`_.
+
+This led to the merge of the three projects.
+
+The original contents of the shade library have been moved into
+``openstack.cloud`` and os-client-config has been moved in to
+``openstack.config``. The next release of shade will be a thin compatibility
+layer that subclasses the objects from ``openstack.cloud`` and provides
+different argument defaults where needed for compat.
+Similarly the next release of os-client-config will be a compat
 layer shim around ``openstack.config``.
 
+.. note::
+
+  The ``openstack.cloud.OpenStackCloud`` object and the
+  ``openstack.connection.Connection`` object are going to be merged. It is
+  recommended to not write any new code which consumes objects from the
+  ``openstack.cloud`` namespace until that merge is complete.
+
+.. _nodepool: https://docs.openstack.org/infra/nodepool/
+.. _Ansible OpenStack Modules: http://docs.ansible.com/ansible/latest/list_of_cloud_modules.html#openstack
+.. _Session: http://docs.python-requests.org/en/master/user/advanced/#session-objects
+.. _python-requests: http://docs.python-requests.org/en/master/
+
+openstack
+=========
+
+List servers using objects configured with the ``clouds.yaml`` file:
+
+.. code-block:: python
+
+    import openstack
+
+    # Initialize and turn on debug logging
+    openstack.enable_logging(debug=True)
+
+    # Initialize cloud
+    conn = openstack.connect(cloud='mordred')
+
+    for server in conn.compute.servers():
+        print(server.to_dict())
+
 openstack.config
 ================
 
@@ -88,10 +138,10 @@ Create a server using objects configured with the ``clouds.yaml`` file:
 
 .. code-block:: python
 
-    import openstack.cloud
+    import openstack
 
     # Initialize and turn on debug logging
-    openstack.cloud.simple_logging(debug=True)
+    openstack.enable_logging(debug=True)
 
     # Initialize cloud
     # Cloud configs are read with openstack.config

SHADE-MERGE-TODO.rst

@@ -72,7 +72,6 @@ shade integration
 * Investigate auto-generating the bulk of shade's API based on introspection of
   SDK objects, leaving only the code with extra special logic in the shade
   layer.
-* Rationalize openstack.util.enable_logging and shade.simple_logging.
 
 Service Proxies
 ---------------

doc/source/user/guides/logging.rst

@@ -1,80 +1,113 @@
+=======
 Logging
 =======
 
-Logging can save you time and effort when developing your code or looking
-for help. If your code is not behaving how you expect it to, enabling and
-configuring logging can quickly give you valuable insight into the root
-cause of the issue. If you need help from the OpenStack community, the
-logs can help the people there assist you.
+.. note:: TODO(shade) This document is written from a shade POV. It needs to
+          be combined with the existing logging guide, but also the logging
+          systems need to be rationalized.
+
+`openstacksdk` uses `Python Logging`_. As `openstacksdk` is a library, it does
+not configure logging handlers automatically, expecting instead for that to be
+the purview of the consuming application.
+
+Simple Usage
+------------
+
+For consumers who just want to get a basic logging setup without thinking
+about it too deeply, there is a helper method. If used, it should be called
+before any other openstacksdk functionality.
+
+.. autofunction:: openstack.enable_logging
+
+.. code-block:: python
 
-.. note:: By default, no logging is done.
+  import openstack
+  openstack.enable_logging()
 
-Enable SDK Logging
-------------------
+The ``stream`` parameter controls the stream where log message are written to.
+It defaults to `sys.stdout` which will result in log messages being written
+to STDOUT. It can be set to another output stream, or to ``None`` to disable
+logging to the console.
+
+The ``path`` parameter sets up logging to log to a file. By default, if
+``path`` is given and ``stream`` is not, logging will only go to ``path``.
 
-To enable logging you use :func:`~openstack.utils.enable_logging`.
+You can combine the ``path`` and ``stream`` parameters to log to both places
+simultaneously.
 
-The ``debug`` parameter controls the logging level. Set ``debug=True`` to
-log debug and higher messages. Set ``debug=False`` to log warning and higher
-messages.
+To log messages to a file called ``openstack.log`` and the console on
+``stdout``:
 
-To log debug and higher messages::
+.. code-block:: python
 
     import sys
     from openstack import utils
 
-    utils.enable_logging(debug=True, stream=sys.stdout)
+    utils.enable_logging(debug=True, path='openstack.log', stream=sys.stdout)
 
-The ``path`` parameter controls the location of a log file. If set, this
-parameter will send log messages to a file using a
-:py:class:`~logging.FileHandler`.
 
-To log messages to a file called ``openstack.log``::
+`openstack.enable_logging` also sets up a few other loggers and
+squelches some warnings or log messages that are otherwise uninteresting or
+unactionable by an openstacksdk user.
 
-    from openstack import utils
+Advanced Usage
+--------------
 
-    utils.enable_logging(debug=True, path='openstack.log')
+`openstacksdk` logs to a set of different named loggers.
 
-The ``stream`` parameter controls the stream where log message are written to.
-If set to ``sys.stdout`` or ``sys.stderr``, this parameter will send log
-messages to that stream using a :py:class:`~logging.StreamHandler`
+Most of the logging is set up to log to the root ``openstack`` logger.
+There are additional sub-loggers that are used at times, primarily so that a
+user can decide to turn on or off a specific type of logging. They are listed
+below.
 
-To log messages to the console on ``stdout``::
+openstack.config
+  Issues pertaining to configuration are logged to the ``openstack.config``
+  logger.
 
-    import sys
-    from openstack import utils
+openstack.task_manager
+  `openstacksdk` uses a Task Manager to perform remote calls. The
+  ``openstack.task_manager`` logger emits messages at the start and end
+  of each Task announcing what it is going to run and then what it ran and
+  how long it took. Logging ``openstack.task_manager`` is a good way to
+  get a trace of external actions `openstacksdk` is taking without full
+  `HTTP Tracing`_.
 
-    utils.enable_logging(debug=True, stream=sys.stdout)
+openstack.iterate_timeout
+  When `openstacksdk` needs to poll a resource, it does so in a loop that waits
+  between iterations and ultimately times out. The
+  ``openstack.iterate_timeout`` logger emits messages for each iteration
+  indicating it is waiting and for how long. These can be useful to see for
+  long running tasks so that one can know things are not stuck, but can also
+  be noisy.
 
-You can combine the ``path`` and ``stream`` parameters to log to both places
-simultaneously.
+openstack.fnmatch
+  `openstacksdk` will try to use `fnmatch`_ on given `name_or_id` arguments.
+  It's a best effort attempt, so pattern misses are logged to
+  ``openstack.fnmatch``. A user may not be intending to use an fnmatch
+  pattern - such as if they are trying to find an image named
+  ``Fedora 24 [official]``, so these messages are logged separately.
 
-To log messages to a file called ``openstack.log`` and the console on
-``stdout``::
+.. _fnmatch: https://pymotw.com/2/fnmatch/
 
-    import sys
-    from openstack import utils
+HTTP Tracing
+------------
 
-    utils.enable_logging(debug=True, path='openstack.log', stream=sys.stdout)
+HTTP Interactions are handled by `keystoneauth`_. If you want to enable HTTP
+tracing while using openstacksdk and are not using `openstack.enable_logging`,
+set the log level of the ``keystoneauth`` logger to ``DEBUG``.
 
+For more information see https://docs.openstack.org/keystoneauth/latest/using-sessions.html#logging
 
-Enable requests Logging
------------------------
+.. _keystoneauth: https://docs.openstack.org/keystoneauth/latest/
 
-The SDK depends on a small number other libraries. Notably, it uses
-`requests <https://pypi.python.org/pypi/requests>`_ for its transport layer.
-To get even more information about the request/response cycle, you enable
-logging of requests the same as you would any other library.
+Python Logging
+--------------
 
-To log messages to the console on ``stdout``::
+Python logging is a standard feature of Python and is documented fully in the
+Python Documentation, which varies by version of Python.
 
-    import logging
-    import sys
+For more information on Python Logging for Python v2, see
+https://docs.python.org/2/library/logging.html.
 
-    logger = logging.getLogger('requests')
-    formatter = logging.Formatter(
-        '%(asctime)s %(levelname)s: %(name)s %(message)s')
-    console = logging.StreamHandler(sys.stdout)
-    console.setFormatter(formatter)
-    logger.setLevel(logging.DEBUG)
-    logger.addHandler(console)
+For more information on Python Logging for Python v3, see
+https://docs.python.org/3/library/logging.html.

doc/source/user/index.rst

@@ -22,8 +22,6 @@ These guides walk you through how to make use of the libraries we provide
 to work with each OpenStack service. If you're looking for a cookbook
 approach, this is where you'll want to begin.
 
-.. TODO(shade) Merge guides/logging and logging
-
 .. toctree::
    :maxdepth: 1
 
@@ -32,7 +30,6 @@ approach, this is where you'll want to begin.
    Connect to an OpenStack Cloud Using a Config File <guides/connect_from_config>
    Using Cloud Abstration Layer <usage>
    Logging <guides/logging>
-   Shade Logging <logging>
    Microversions <microversions>
    Baremetal <guides/baremetal>
    Block Storage <guides/block_storage>