-
Notifications
You must be signed in to change notification settings - Fork 96
Replace Order with an attrs dataclass #403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
92bc03a
d79402d
8013136
ebf60f5
9d340aa
fd2545b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -68,8 +68,9 @@ def orders(ctx, base_url): | |
async def list(ctx, state, limit, pretty): | ||
'''List orders''' | ||
async with orders_client(ctx) as cl: | ||
orders = await cl.list_orders(state=state, limit=limit, as_json=True) | ||
orders = await cl.list_orders(state=state, limit=limit) | ||
|
||
orders = [order.to_dict() for order in orders] | ||
echo_json(orders, pretty) | ||
|
||
|
||
|
@@ -87,7 +88,7 @@ async def get(ctx, order_id, pretty): | |
async with orders_client(ctx) as cl: | ||
order = await cl.get_order(str(order_id)) | ||
|
||
echo_json(order.json, pretty) | ||
echo_json(order.to_dict(), pretty) | ||
|
||
|
||
@orders.command() | ||
|
@@ -263,4 +264,4 @@ async def create(ctx, | |
async with orders_client(ctx) as cl: | ||
order = await cl.create_order(request) | ||
|
||
echo_json(order.json, pretty) | ||
echo_json(order.to_dict(), pretty) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we're encoding to JSON as late as possible this can just be an ordinary function and not a property that we might cache or whatever. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The property There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right. We're only going to call Also I think |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -12,6 +12,7 @@ | |
# License for the specific language governing permissions and limitations under | ||
# the License. | ||
"""Functionality for interacting with the orders api""" | ||
|
||
import asyncio | ||
import json | ||
import logging | ||
|
@@ -20,10 +21,17 @@ | |
import typing | ||
import uuid | ||
|
||
import arrow | ||
|
||
from .. import exceptions | ||
from ..constants import PLANET_BASE_URL | ||
from ..http import Session | ||
from ..models import Order, Orders, Request, Response, StreamingBody | ||
from ..models import (Order, | ||
OrderComponent, | ||
Orders, | ||
Request, | ||
Response, | ||
StreamingBody) | ||
sgillies marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
BASE_URL = f'{PLANET_BASE_URL}/compute/ops' | ||
STATS_PATH = '/stats/orders/v2' | ||
|
@@ -94,6 +102,45 @@ def _stats_url(self): | |
def _request(self, url, method, data=None, params=None, json=None): | ||
return Request(url, method=method, data=data, params=params, json=json) | ||
|
||
def _order_factory(self, contents: dict) -> Order: | ||
"""Convert API response contents to an instance of Order.""" | ||
links = contents.get("_links", []) | ||
results = [] | ||
|
||
for res in links.get("results", []): | ||
expires_at = res.get("expires_at", None) | ||
if expires_at: | ||
expires = arrow.get(expires_at).datetime | ||
else: | ||
expires = None | ||
|
||
results.append( | ||
OrderComponent(delivery=res.get("delivery", None), | ||
name=res.get("name", None), | ||
location=res.get("location", None), | ||
expires=expires)) | ||
|
||
created_on = contents.get("created_on", None) | ||
if created_on: | ||
created = arrow.get(created_on).datetime | ||
else: | ||
created = None | ||
|
||
last_modified = contents.get("last_modified", None) | ||
if last_modified: | ||
last_modified = arrow.get(last_modified).datetime | ||
else: | ||
last_modified = None | ||
|
||
return Order(id=contents["id"], | ||
state=contents["state"], | ||
results=results, | ||
created=created, | ||
last_modified=last_modified, | ||
last_message=contents["last_message"], | ||
error_hints=contents["error_hints"], | ||
Comment on lines
+138
to
+141
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. New attributes that fully cover the API-provided and dynamic parts of an order description. |
||
request=contents) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
async def _do_request(self, request: Request) -> Response: | ||
'''Submit a request and get response. | ||
|
||
|
@@ -152,7 +199,7 @@ async def create_order(self, request: dict) -> str: | |
pass | ||
raise exceptions.BadQuery(msg) | ||
|
||
order = Order(resp.json()) | ||
order = self._order_factory(resp.json()) | ||
return order | ||
|
||
async def get_order(self, order_id: str) -> Order: | ||
|
@@ -180,7 +227,7 @@ async def get_order(self, order_id: str) -> Order: | |
msg = msg_json['message'] | ||
raise exceptions.MissingResource(msg) | ||
|
||
order = Order(resp.json()) | ||
order = self._order_factory(resp.json()) | ||
return order | ||
|
||
async def cancel_order(self, order_id: str) -> Response: | ||
|
@@ -303,7 +350,7 @@ async def download_order(self, | |
planet.exceptions.APIException: On API error. | ||
""" | ||
order = await self.get_order(order_id) | ||
locations = order.locations | ||
locations = [component.location for component in order.results] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consequence of adding a new OrderComponent class (alluded to in https://developers.planet.com/docs/orders/reference/#operation/getOrder). |
||
LOGGER.info( | ||
f'downloading {len(locations)} assets from order {order_id}') | ||
filenames = [ | ||
|
@@ -384,14 +431,12 @@ async def poll(self, | |
async def list_orders( | ||
self, | ||
state: str = None, | ||
limit: int = None, | ||
as_json: bool = False) -> typing.Union[typing.List[Order], dict]: | ||
limit: int = None) -> typing.Union[typing.List[Order], dict]: | ||
"""Get all order requests. | ||
|
||
Parameters: | ||
state: Filter orders to given state. | ||
limit: Limit orders to given limit. | ||
as_json: Return orders as a json dict. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This method will be much easier to maintain if it only returns a list of Orders. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. agreed! |
||
|
||
Returns: | ||
User orders that match the query | ||
|
@@ -407,17 +452,15 @@ async def list_orders( | |
params = None | ||
|
||
orders = await self._get_orders(url, params=params, limit=limit) | ||
|
||
if as_json: | ||
ret = [o.json async for o in orders] | ||
else: | ||
ret = [o async for o in orders] | ||
return ret | ||
return [order async for order in orders] | ||
|
||
async def _get_orders(self, url, params=None, limit=None): | ||
request = self._request(url, 'GET', params=params) | ||
|
||
return Orders(request, self._do_request, limit=limit) | ||
return Orders(request, | ||
self._do_request, | ||
self._order_factory, | ||
limit=limit) | ||
|
||
@staticmethod | ||
def _check_state(state): | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,13 +14,14 @@ | |
"""Manage data for requests and responses.""" | ||
import copy | ||
from datetime import datetime | ||
import json | ||
import logging | ||
import mimetypes | ||
import random | ||
import re | ||
import string | ||
import typing | ||
|
||
from attrs import define | ||
import httpx | ||
from tqdm.asyncio import tqdm | ||
|
||
|
@@ -265,6 +266,64 @@ def _get_random_filename(content_type=None): | |
return name | ||
|
||
|
||
@define | ||
class OrderComponent: | ||
"""Description of a component, or part, of a Planet data order. | ||
|
||
Attributes: | ||
delivery (str): The current state of this order request | ||
component. | ||
expires (datetime): When the download link (if present) expires. | ||
location (str): The (optional) download link for this asset. | ||
name (str): The name of the component. | ||
|
||
""" | ||
delivery: str | ||
expires: datetime | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jreiberkyle the API response has |
||
location: str | ||
name: str | ||
|
||
|
||
@define | ||
class Order: | ||
"""Description of a Planet data order. | ||
|
||
There are two kinds of information in an order description: the | ||
product and delivery criteria chosen by the user, which is called | ||
the "request"; and several attributes describing the state of order | ||
processing which are determined by the Planet Orders API. | ||
|
||
Attributes: | ||
created (datetime): The UTC date this Order request was created. | ||
Provided by the Orders API. | ||
error_hints (list of str): Hints related to any reported error. | ||
API provided. | ||
id (str): A UUID to uniquely identify this Order request. API | ||
provided. | ||
last_message (str): Some info on the current order state. API | ||
provided. | ||
last_modified (str): The UTC date this Order request was last | ||
modified. API provided. | ||
request (dict): Product and delivery criteria. User provided. | ||
results (list of OrderComponent): The canonical links to this | ||
Order's results. API provided. | ||
state (str): The current state of this Order request. API | ||
provided. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All the attribute descriptions come straight from https://developers.planet.com/docs/orders/reference/#operation/getOrder. |
||
|
||
""" | ||
created: datetime | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same for |
||
error_hints: typing.List[str] | ||
id: str | ||
last_message: str | ||
last_modified: datetime | ||
request: dict | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
results: typing.List[OrderComponent] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Previously, Order exposed results and locations as separate lists. Here we only expose a list of OrderComponents, which is a simplification. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Order is only a data class now. I think this is a great match for its role in our information model: it's only a snapshot of the state of an order, the truth of which exists entirely in our API server. |
||
state: str | ||
|
||
def to_dict(self): | ||
return self.request | ||
|
||
|
||
class Paged(): | ||
'''Asynchronous iterator over results in a paged resource from the Planet | ||
server. | ||
|
@@ -349,71 +408,16 @@ def _next_link(self, page): | |
return next_link | ||
|
||
|
||
class Order(): | ||
'''Managing description of an order returned from Orders API. | ||
|
||
:param data: Response json describing order | ||
:type data: dict | ||
''' | ||
LINKS_KEY = '_links' | ||
RESULTS_KEY = 'results' | ||
LOCATION_KEY = 'location' | ||
|
||
def __init__(self, data): | ||
self.data = data | ||
|
||
def __str__(self): | ||
return "<Order> " + json.dumps(self.data) | ||
|
||
@property | ||
def results(self): | ||
'''Results for each item in order. | ||
|
||
:return: result for each item in order | ||
:rtype: list of dict | ||
''' | ||
links = self.data[self.LINKS_KEY] | ||
results = links.get(self.RESULTS_KEY, None) | ||
return results | ||
|
||
@property | ||
def locations(self): | ||
'''Download locations for order results. | ||
|
||
:return: download locations in order | ||
:rtype: list of str | ||
''' | ||
return list(r[self.LOCATION_KEY] for r in self.results) | ||
|
||
@property | ||
def state(self): | ||
'''State of the order. | ||
|
||
:return: state of order | ||
:rtype: str | ||
''' | ||
return self.data['state'] | ||
|
||
@property | ||
def id(self): | ||
'''ID of the order. | ||
|
||
:return: id of order | ||
:rtype: str | ||
''' | ||
return self.data['id'] | ||
|
||
@property | ||
def json(self): | ||
return self.data | ||
|
||
|
||
class Orders(Paged): | ||
'''Asynchronous iterator over Orders from a paged response describing | ||
orders.''' | ||
LINKS_KEY = '_links' | ||
NEXT_KEY = 'next' | ||
ITEMS_KEY = 'orders' | ||
|
||
def __init__(self, request, do_request_fcn, order_factory, limit=None): | ||
super().__init__(request, do_request_fcn, limit=limit) | ||
self._order_factory = order_factory | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We inject a factory for Orders. |
||
|
||
async def __anext__(self): | ||
return Order(await super().__anext__()) | ||
return self._order_factory(await super().__anext__()) |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,6 +23,7 @@ | |
continue | ||
|
||
install_requires = [ | ||
'arrow', | ||
'click>=8.0.0', | ||
'httpx==0.16.1', | ||
'shapely>=1.7.1', | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -164,20 +164,6 @@ async def test_list_orders_limit(order_descriptions, session): | |
assert oids == ['oid1'] | ||
|
||
|
||
@respx.mock | ||
@pytest.mark.asyncio | ||
async def test_list_orders_asjson(order_descriptions, session): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No longer needed. |
||
order1, order2, order3 = order_descriptions | ||
|
||
page1_response = {"_links": {"_self": "string"}, "orders": [order1]} | ||
mock_resp1 = httpx.Response(HTTPStatus.OK, json=page1_response) | ||
respx.get(TEST_ORDERS_URL).return_value = mock_resp1 | ||
|
||
cl = OrdersClient(session, base_url=TEST_URL) | ||
orders = await cl.list_orders(as_json=True) | ||
assert orders[0]['id'] == 'oid1' | ||
|
||
|
||
@respx.mock | ||
@pytest.mark.asyncio | ||
async def test_create_order(oid, order_description, order_request, session): | ||
|
@@ -187,7 +173,8 @@ async def test_create_order(oid, order_description, order_request, session): | |
cl = OrdersClient(session, base_url=TEST_URL) | ||
order = await cl.create_order(order_request) | ||
|
||
assert order.json == order_description | ||
assert order.id == order_description["id"] | ||
assert order.state == order_description["state"] | ||
|
||
|
||
@respx.mock | ||
|
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jreiberkyle let's keep it so only the outermost layer of our system, the CLI commands, are concerned about JSON. On the inside, let's use instances of the classes in models.py: Object, ObjectComponent, etc. My reasoning is based on an analogy to Ned Batchelder's "Unicode Sandwich".
Also, methods that return one type of thing or another type based on an keyword argument are harder to maintain and refactor in the long run. We'll feel good about a single return type in 6 months or a year, I predict.
@tschaub I'm cc'ing you on this since you're getting back into Python programming. I hope the STAC ecosystem agrees with me about the JSON 🥪 😄
Decode JSON to data classes as soon as possible. Use data classes throughout the program. Encode back to JSON as late as possible.
cc @mkshah605
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh. This is an interesting proposal and I can understand it, especially the analogy, and I still wonder about the tradeoff between working with JSON internally and working with data structures which are pretty much just wrapped JSON. Does the Python library user prefer the user experience of managing data structures vs managing JSON? I don't know the answer, though my original assumption was that they would prefer JSON. I would love to learn more about this ux decision in the wild. I will check out the python libraries associated with aws and gcloud - any others I could check?
This I full-heartedly agree with 😄