docs: clarify where StoredState is stored, and the upgrade behaviour (#…

…1416)

Adjustments to the `StoredState` documentation:

* For `StoredDict`, `StoredList`, and `StoredSet`, note that charms
would not normally create these directly and would use `StoredState`
instead.
* Explain where the data is stored (there is one case that it doesn't
mention, where there is already a unit-state DB locally, but I think
that complicates things too much to specify here).
* Explain what happens when the charm is upgraded and recommend using a
peer relation over stored state for K8s charms that want data preserved
across upgrades.

As a related drive-by: remove the `jujuremoved` tag for
`use_juju_for_storage`, since this won't actually be going away with
Juju 4.0.

Fixes #1270

ops/__init__.py

@@ -371,7 +371,4 @@ class SomeCharm(ops.CharmBase): ...
         Podspec charms that haven't previously used local storage and that
         are running on a new enough Juju default to controller-side storage,
         and local storage otherwise.
-
-.. jujuremoved:: 4.0
-    The ``use_juju_for_storage`` argument is not available from Juju 4.0
 """

ops/framework.py

@@ -1132,14 +1132,9 @@ def set_default(self, **kwargs: Any):
 class StoredState:
     """A class used to store data the charm needs, persisted across invocations.
 
-    Example::
-
-        class MyClass(ops.Object):
-            _stored = ops.StoredState()
-
     Instances of ``MyClass`` can transparently save state between invocations by
     setting attributes on ``_stored``. Initial state should be set with
-    ``set_default`` on the bound object, that is::
+    ``set_default`` on the bound object; for example::
 
         class MyClass(ops.Object):
             _stored = ops.StoredState()
@@ -1152,6 +1147,16 @@ def __init__(self, parent, key):
             def _on_seen(self, event):
                 self._stored.seen.add(event.uuid)
 
+    Data is stored alongside the charm (in the charm container for Kubernetes
+    sidecar charms, and on the machine for machine charms). The exceptions are
+    two deprecated cases: Kubernetes podspec charms, and charms explicitly
+    passing `True` for `use_juju_for_storage` when running :meth:`ops.main`.
+
+    For machine charms, charms are upgraded in-place on the machine, so the data
+    is preserved. For Kubernetes sidecar charms, when the charm is upgraded, the
+    pod is replaced, so any data is lost. When data should be preserved across
+    upgrades, Kubernetes sidecar charms should use a peer-relation for the data
+    instead of `StoredState`.
     """
 
     def __init__(self):
@@ -1245,7 +1250,12 @@ def _wrapped_repr(obj: '_StoredObject') -> str:
 
 
 class StoredDict(typing.MutableMapping[Hashable, Any]):
-    """A dict-like object that uses the StoredState as backend."""
+    """A dict-like object that uses the StoredState as backend.
+
+    Charms are not expected to use this class directly. Adding a
+    :class:`StoredState` attribute to a charm class will automatically use this
+    class to store dictionaries.
+    """
 
     def __init__(self, stored_data: StoredStateData, under: Dict[Hashable, Any]):
         self._stored_data = stored_data
@@ -1280,7 +1290,12 @@ def __eq__(self, other: Any):
 
 
 class StoredList(typing.MutableSequence[Any]):
-    """A list-like object that uses the StoredState as backend."""
+    """A list-like object that uses the StoredState as backend.
+
+    Charms are not expected to use this class directly. Adding a
+    :class:`StoredState` attribute to a charm class will automatically use this
+    class to store lists.
+    """
 
     def __init__(self, stored_data: StoredStateData, under: List[Any]):
         self._stored_data = stored_data
@@ -1354,7 +1369,12 @@ def __ge__(self, other: Any):
 
 
 class StoredSet(typing.MutableSet[Any]):
-    """A set-like object that uses the StoredState as backend."""
+    """A set-like object that uses the StoredState as backend.
+
+    Charms are not expected to use this class directly. Adding a
+    :class:`StoredState` attribute to a charm class will automatically use this
+    class to store sets.
+    """
 
     def __init__(self, stored_data: StoredStateData, under: Set[Any]):
         self._stored_data = stored_data