Sporadic Unauthenticated error on App Engine

[jacobg]

On App Engine Python 3.9, the Python Datastore client is raising sporadic Unauthenticated errors. By far, most calls are successful. Here is the stacktrace:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 50, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

The above exception (<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."
        debug_error_string = "{"created":"@1660750490.925409568","description":"Error received from peer ipv4:[74.125.70.95:443](http://74.125.70.95:443/)","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.","grpc_status":16}"
>) was the direct cause of the following exception:

for entity in query.fetch(limit=limit, offset=offset):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 208, in _items_iter
    for page in self._page_iter(increment=False):
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 244, in _page_iter
    page = self._next_page()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/datastore/query.py", line 627, in _next_page
    response_pb = self.client._datastore_api.run_query(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/datastore_v1/services/datastore/client.py", line 579, in run_query
    response = rpc(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 52, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc

Exception Type: Unauthenticated
Exception Value: 401 Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project. [reason: "CREDENTIALS_MISSING"
domain: "[googleapis.com](http://googleapis.com/)"
metadata {
  key: "method"
  value: "google.datastore.v1.Datastore.RunQuery"
}
metadata {
  key: "service"
  value: "[datastore.googleapis.com](http://datastore.googleapis.com/)"
}
]
Request data not supplied

This isn't usually what transient errors look like (e.g., 503). Is that an expected transient type of error?