Skip to content

Replace Order with an attrs dataclass #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions planet/cli/orders.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,9 @@ def orders(ctx, base_url):
async def list(ctx, state, limit, pretty):
'''List orders'''
async with orders_client(ctx) as cl:
orders = await cl.list_orders(state=state, limit=limit, as_json=True)
orders = await cl.list_orders(state=state, limit=limit)

orders = [order.to_dict() for order in orders]
Copy link
Contributor Author

@sgillies sgillies Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreiberkyle let's keep it so only the outermost layer of our system, the CLI commands, are concerned about JSON. On the inside, let's use instances of the classes in models.py: Object, ObjectComponent, etc. My reasoning is based on an analogy to Ned Batchelder's "Unicode Sandwich".

As we saw with Fact of Life #1, the data coming into and going out of your program must be bytes. But you don’t need to deal with bytes on the inside of your program. The best strategy is to decode incoming bytes as soon as possible, producing unicode. You use unicode throughout your program, and then when outputting data, encode it to bytes as late as possible.

This creates a Unicode sandwich: bytes on the outside, Unicode on the inside.

Also, methods that return one type of thing or another type based on an keyword argument are harder to maintain and refactor in the long run. We'll feel good about a single return type in 6 months or a year, I predict.

@tschaub I'm cc'ing you on this since you're getting back into Python programming. I hope the STAC ecosystem agrees with me about the JSON 🥪 😄

Decode JSON to data classes as soon as possible. Use data classes throughout the program. Encode back to JSON as late as possible.

cc @mkshah605

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh. This is an interesting proposal and I can understand it, especially the analogy, and I still wonder about the tradeoff between working with JSON internally and working with data structures which are pretty much just wrapped JSON. Does the Python library user prefer the user experience of managing data structures vs managing JSON? I don't know the answer, though my original assumption was that they would prefer JSON. I would love to learn more about this ux decision in the wild. I will check out the python libraries associated with aws and gcloud - any others I could check?

Also, methods that return one type of thing or another type based on an keyword argument are harder to maintain and refactor in the long run. We'll feel good about a single return type in 6 months or a year, I predict.

This I full-heartedly agree with 😄

echo_json(orders, pretty)


Expand All @@ -87,7 +88,7 @@ async def get(ctx, order_id, pretty):
async with orders_client(ctx) as cl:
order = await cl.get_order(str(order_id))

echo_json(order.json, pretty)
echo_json(order.to_dict(), pretty)


@orders.command()
Expand Down Expand Up @@ -263,4 +264,4 @@ async def create(ctx,
async with orders_client(ctx) as cl:
order = await cl.create_order(request)

echo_json(order.json, pretty)
echo_json(order.to_dict(), pretty)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're encoding to JSON as late as possible this can just be an ordinary function and not a property that we might cache or whatever.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The property json could be a function that calls e.g. to_dict(), so it seems the change would be based on whether it improves ux, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. We're only going to call Order.to_dict() once in a CLI command, just before we print the JSON-encoded result, and I think that'll very likely be true in user code.

Also I think json.dumps(order.to_dict()) makes more sense that json.dumps(order.json). The second one reads like we're JSON-encoding twice.

71 changes: 57 additions & 14 deletions planet/clients/orders.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# License for the specific language governing permissions and limitations under
# the License.
"""Functionality for interacting with the orders api"""

import asyncio
import json
import logging
Expand All @@ -20,10 +21,17 @@
import typing
import uuid

import arrow

from .. import exceptions
from ..constants import PLANET_BASE_URL
from ..http import Session
from ..models import Order, Orders, Request, Response, StreamingBody
from ..models import (Order,
OrderComponent,
Orders,
Request,
Response,
StreamingBody)

BASE_URL = f'{PLANET_BASE_URL}/compute/ops'
STATS_PATH = '/stats/orders/v2'
Expand Down Expand Up @@ -94,6 +102,45 @@ def _stats_url(self):
def _request(self, url, method, data=None, params=None, json=None):
return Request(url, method=method, data=data, params=params, json=json)

def _order_factory(self, contents: dict) -> Order:
"""Convert API response contents to an instance of Order."""
links = contents.get("_links", [])
results = []

for res in links.get("results", []):
expires_at = res.get("expires_at", None)
if expires_at:
expires = arrow.get(expires_at).datetime
else:
expires = None

results.append(
OrderComponent(delivery=res.get("delivery", None),
name=res.get("name", None),
location=res.get("location", None),
expires=expires))

created_on = contents.get("created_on", None)
if created_on:
created = arrow.get(created_on).datetime
else:
created = None

last_modified = contents.get("last_modified", None)
if last_modified:
last_modified = arrow.get(last_modified).datetime
else:
last_modified = None

return Order(id=contents["id"],
state=contents["state"],
results=results,
created=created,
last_modified=last_modified,
last_message=contents["last_message"],
error_hints=contents["error_hints"],
Comment on lines +138 to +141
Copy link
Contributor Author

@sgillies sgillies Mar 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New attributes that fully cover the API-provided and dynamic parts of an order description.

request=contents)
Copy link
Contributor Author

@sgillies sgillies Mar 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

request is the user-provided order and delivery criteria, the data that we post to the API when we create an order.


async def _do_request(self, request: Request) -> Response:
'''Submit a request and get response.

Expand Down Expand Up @@ -152,7 +199,7 @@ async def create_order(self, request: dict) -> str:
pass
raise exceptions.BadQuery(msg)

order = Order(resp.json())
order = self._order_factory(resp.json())
return order

async def get_order(self, order_id: str) -> Order:
Expand Down Expand Up @@ -180,7 +227,7 @@ async def get_order(self, order_id: str) -> Order:
msg = msg_json['message']
raise exceptions.MissingResource(msg)

order = Order(resp.json())
order = self._order_factory(resp.json())
return order

async def cancel_order(self, order_id: str) -> Response:
Expand Down Expand Up @@ -303,7 +350,7 @@ async def download_order(self,
planet.exceptions.APIException: On API error.
"""
order = await self.get_order(order_id)
locations = order.locations
locations = [component.location for component in order.results]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consequence of adding a new OrderComponent class (alluded to in https://developers.planet.com/docs/orders/reference/#operation/getOrder).

LOGGER.info(
f'downloading {len(locations)} assets from order {order_id}')
filenames = [
Expand Down Expand Up @@ -384,14 +431,12 @@ async def poll(self,
async def list_orders(
self,
state: str = None,
limit: int = None,
as_json: bool = False) -> typing.Union[typing.List[Order], dict]:
limit: int = None) -> typing.Union[typing.List[Order], dict]:
"""Get all order requests.

Parameters:
state: Filter orders to given state.
limit: Limit orders to given limit.
as_json: Return orders as a json dict.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method will be much easier to maintain if it only returns a list of Orders.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed!


Returns:
User orders that match the query
Expand All @@ -407,17 +452,15 @@ async def list_orders(
params = None

orders = await self._get_orders(url, params=params, limit=limit)

if as_json:
ret = [o.json async for o in orders]
else:
ret = [o async for o in orders]
return ret
return [order async for order in orders]

async def _get_orders(self, url, params=None, limit=None):
request = self._request(url, 'GET', params=params)

return Orders(request, self._do_request, limit=limit)
return Orders(request,
self._do_request,
self._order_factory,
limit=limit)

@staticmethod
def _check_state(state):
Expand Down
126 changes: 65 additions & 61 deletions planet/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,14 @@
"""Manage data for requests and responses."""
import copy
from datetime import datetime
import json
import logging
import mimetypes
import random
import re
import string
import typing

from attrs import define
import httpx
from tqdm.asyncio import tqdm

Expand Down Expand Up @@ -265,6 +266,64 @@ def _get_random_filename(content_type=None):
return name


@define
class OrderComponent:
"""Description of a component, or part, of a Planet data order.

Attributes:
delivery (str): The current state of this order request
component.
expires (datetime): When the download link (if present) expires.
location (str): The (optional) download link for this asset.
name (str): The name of the component.

"""
delivery: str
expires: datetime
Copy link
Contributor Author

@sgillies sgillies Mar 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreiberkyle the API response has expires_at, but that's an idiosyncracy. I think users will be less surprised to see expires.

location: str
name: str


@define
class Order:
"""Description of a Planet data order.

There are two kinds of information in an order description: the
product and delivery criteria chosen by the user, which is called
the "request"; and several attributes describing the state of order
processing which are determined by the Planet Orders API.

Attributes:
created (datetime): The UTC date this Order request was created.
Provided by the Orders API.
error_hints (list of str): Hints related to any reported error.
API provided.
id (str): A UUID to uniquely identify this Order request. API
provided.
last_message (str): Some info on the current order state. API
provided.
last_modified (str): The UTC date this Order request was last
modified. API provided.
request (dict): Product and delivery criteria. User provided.
results (list of OrderComponent): The canonical links to this
Order's results. API provided.
state (str): The current state of this Order request. API
provided.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the attribute descriptions come straight from https://developers.planet.com/docs/orders/reference/#operation/getOrder.


"""
created: datetime
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error_hints: typing.List[str]
id: str
last_message: str
last_modified: datetime
request: dict
Copy link
Contributor Author

@sgillies sgillies Mar 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

request primarily exists to contain the user-provided order and delivery criteria. In this implementation it will also contain the entire order description sent back by the API because it's simpler to just do that than strip out API-provided items that are also exposed by class attributes.

results: typing.List[OrderComponent]
Copy link
Contributor Author

@sgillies sgillies Mar 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, Order exposed results and locations as separate lists. Here we only expose a list of OrderComponents, which is a simplification.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Order is only a data class now. I think this is a great match for its role in our information model: it's only a snapshot of the state of an order, the truth of which exists entirely in our API server.

state: str

def to_dict(self):
return self.request


class Paged():
'''Asynchronous iterator over results in a paged resource from the Planet
server.
Expand Down Expand Up @@ -349,71 +408,16 @@ def _next_link(self, page):
return next_link


class Order():
'''Managing description of an order returned from Orders API.

:param data: Response json describing order
:type data: dict
'''
LINKS_KEY = '_links'
RESULTS_KEY = 'results'
LOCATION_KEY = 'location'

def __init__(self, data):
self.data = data

def __str__(self):
return "<Order> " + json.dumps(self.data)

@property
def results(self):
'''Results for each item in order.

:return: result for each item in order
:rtype: list of dict
'''
links = self.data[self.LINKS_KEY]
results = links.get(self.RESULTS_KEY, None)
return results

@property
def locations(self):
'''Download locations for order results.

:return: download locations in order
:rtype: list of str
'''
return list(r[self.LOCATION_KEY] for r in self.results)

@property
def state(self):
'''State of the order.

:return: state of order
:rtype: str
'''
return self.data['state']

@property
def id(self):
'''ID of the order.

:return: id of order
:rtype: str
'''
return self.data['id']

@property
def json(self):
return self.data


class Orders(Paged):
'''Asynchronous iterator over Orders from a paged response describing
orders.'''
LINKS_KEY = '_links'
NEXT_KEY = 'next'
ITEMS_KEY = 'orders'

def __init__(self, request, do_request_fcn, order_factory, limit=None):
super().__init__(request, do_request_fcn, limit=limit)
self._order_factory = order_factory
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We inject a factory for Orders.


async def __anext__(self):
return Order(await super().__anext__())
return self._order_factory(await super().__anext__())
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
continue

install_requires = [
'arrow',
'click>=8.0.0',
'httpx==0.16.1',
'shapely>=1.7.1',
Expand Down
17 changes: 2 additions & 15 deletions tests/integration/test_orders_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -164,20 +164,6 @@ async def test_list_orders_limit(order_descriptions, session):
assert oids == ['oid1']


@respx.mock
@pytest.mark.asyncio
async def test_list_orders_asjson(order_descriptions, session):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer needed.

order1, order2, order3 = order_descriptions

page1_response = {"_links": {"_self": "string"}, "orders": [order1]}
mock_resp1 = httpx.Response(HTTPStatus.OK, json=page1_response)
respx.get(TEST_ORDERS_URL).return_value = mock_resp1

cl = OrdersClient(session, base_url=TEST_URL)
orders = await cl.list_orders(as_json=True)
assert orders[0]['id'] == 'oid1'


@respx.mock
@pytest.mark.asyncio
async def test_create_order(oid, order_description, order_request, session):
Expand All @@ -187,7 +173,8 @@ async def test_create_order(oid, order_description, order_request, session):
cl = OrdersClient(session, base_url=TEST_URL)
order = await cl.create_order(order_request)

assert order.json == order_description
assert order.id == order_description["id"]
assert order.state == order_description["state"]


@respx.mock
Expand Down
Loading