Converting Logging client->list_entries to iterator. #2636

dhermes · 2016-10-28T04:57:01Z

@waprin @jonparrott I just realized how weird the logging HTTP/JSON API is. For /entries:list, POST is used instead of GET, and the "query params" come in the payload instead of as query params.

~~Just as #2633 started, I'm sending this out without fixing unit tests to get it in the hands of reviewers. The unit tests may take me a non-trivial time, but I am working on them.~~

theacodes · 2016-10-28T05:51:55Z

iirc logging has some very strange behavior with listing entries, I think @waprin is pretty familiar with it.

waprin · 2016-10-28T18:24:41Z

Yes specifically it uses the page tokens as a progress indicator, not just as a means of breaking results into pages. This means you can get 0 results on first page, but then 10 results on the next page, so you really always want to check the page token. This is especially true for slower queries, which are most of them without an index e.g. logName= .

dhermes · 2016-10-28T18:32:18Z

@tseaver @daspecster PTAL

dhermes · 2016-10-28T18:38:19Z

@waprin So what are the implications for using an Iterator? It'll just take a long time consuming empty pages until the results are available?

waprin · 2016-10-28T20:57:40Z

@dhermes yeah, if anything I'm thinking this is a very good change because with the iterator people are less likely to get no results on the first page and think there are no results.

theacodes · 2016-10-28T20:58:17Z

@waprin is there a possibility the iterator will never finish?

core/google/cloud/iterator.py

-            method='GET', path=self.path,
-            query_params=self._get_query_params())
+        params = self._get_query_params()
+        if self._HTTP_METHOD == 'GET':


core/unit_tests/test_iterator.py

+            'data': {},
+        })
+
+    def test__get_next_page_bad_http_method(self):


docs/logging-usage.rst

@@ -68,8 +68,7 @@ Fetch entries for the default project.

   >>> from google.cloud import logging
   >>> client = logging.Client()
-   >>> entries, token = client.list_entries()  # API call
-   >>> for entry in entries:
+   >>> for entry in client.list_entries():  # API call(s)


logging/google/cloud/logging/_helpers.py

+    """Detect correct entry type from resource and instantiate.
+
+    :type resource: dict
+    :param resource: one entry resource from API response


logging/google/cloud/logging/_helpers.py

+    elif 'protoPayload' in resource:
+        return ProtobufEntry.from_api_repr(resource, client, loggers)
+
+    raise ValueError('Cannot parse log entry resource')


logging/unit_tests/test__gax.py

-        entry_pb = LogEntry(log_name=self.LOG_NAME,
+        timestamp = datetime.datetime.utcnow().replace(tzinfo=UTC)
+        timestamp_pb = _datetime_to_pb_timestamp(timestamp)
+        entry_pb = LogEntry(log_name=self.LOG_PATH,


core/google/cloud/iterator.py

+            return self.client.connection.api_request(
+                method=self._HTTP_METHOD,
+                path=self.path,
+                data=params)


waprin · 2016-10-31T16:51:51Z

@jonparrott can't think of a way

dhermes · 2016-10-31T16:53:38Z

@tseaver LGTY to merge?

dhermes · 2016-10-31T17:15:41Z

As discussed with @daspecster, going to make the docstring updates in a follow-up PR

Converting Logging client->list_entries to iterator.

Converting Logging client->list_entries to iterator.

08fda01

dhermes added api: core api: logging Issues related to the Cloud Logging API. labels Oct 28, 2016

dhermes assigned daspecster and tseaver Oct 28, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Oct 28, 2016

Adding unit tests for HTTPIterator GET/POST switching.

bc0c3c4

Adding unit tests for Logging list_entries() change.

e38908e

daspecster reviewed Oct 29, 2016

View reviewed changes

tseaver reviewed Oct 31, 2016

View reviewed changes

core/google/cloud/iterator.py

return self.client.connection.api_request(

method=self._HTTP_METHOD,

path=self.path,

data=params)

This comment was marked as spam.

Sign in to view

dhermes mentioned this pull request Oct 31, 2016

AppVeyor build currently failed due to timestamp format zero padding #2650

Closed

theacodes approved these changes Oct 31, 2016

View reviewed changes

dhermes merged commit 016415a into googleapis:master Oct 31, 2016

dhermes deleted the logging-iterators-1 branch October 31, 2016 17:15

dhermes added a commit to dhermes/google-cloud-python that referenced this pull request Oct 31, 2016

Docstring update follow-up from PR googleapis#2636.

9a3e3b2

dhermes added a commit to dhermes/google-cloud-python that referenced this pull request Oct 31, 2016

Docstring update follow-up from PR googleapis#2636.

d23c1f8

dhermes added a commit to dhermes/google-cloud-python that referenced this pull request Oct 31, 2016

Docstring update follow-up from PR googleapis#2636.

45bf564

dhermes mentioned this pull request Nov 14, 2016

Upgrading core to version to 0.21.0. #2733

Merged

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017

Merge pull request googleapis#2636 from dhermes/logging-iterators-1

b5902a0

Converting Logging client->list_entries to iterator.

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017

Docstring update follow-up from PR googleapis#2636.

c149782

theacodes unassigned daspecster Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Converting Logging client->list_entries to iterator. #2636

Converting Logging client->list_entries to iterator. #2636

dhermes commented Oct 28, 2016 •

edited

Loading

theacodes commented Oct 28, 2016

waprin commented Oct 28, 2016

dhermes commented Oct 28, 2016

dhermes commented Oct 28, 2016

waprin commented Oct 28, 2016

theacodes commented Oct 28, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

waprin commented Oct 31, 2016

dhermes commented Oct 31, 2016

dhermes commented Oct 31, 2016

Converting Logging client->list_entries to iterator. #2636

Converting Logging client->list_entries to iterator. #2636

Conversation

dhermes commented Oct 28, 2016 • edited Loading

theacodes commented Oct 28, 2016

waprin commented Oct 28, 2016

dhermes commented Oct 28, 2016

dhermes commented Oct 28, 2016

waprin commented Oct 28, 2016

theacodes commented Oct 28, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

waprin commented Oct 31, 2016

dhermes commented Oct 31, 2016

dhermes commented Oct 31, 2016

dhermes commented Oct 28, 2016 •

edited

Loading