Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated order delivery config for zipped orders #989

Merged
merged 3 commits into from
Jul 17, 2023

Conversation

JSee98
Copy link
Contributor

@JSee98 JSee98 commented Jul 12, 2023

Related Issue(s):

Closes #963

Proposed Changes:

For inclusion in changelog (if applicable):

  1. Orders now deliver zipped to cloud storage, even when archive_filename and single_archive are undefined.

Not intended for changelog:

Diff of User Interface

Old behavior:
In order_request.delivery(), if archive_type was set to zip, but archive_filename and single_archive were both undefined, data was not delivered zipped.

New behavior:
Now, in order_request.delivery(), if archive_type is set to zip, but archive_filename and single_archive are both undefined, data will deliver as zipped.

PR Checklist:

  • This PR is as small and focused as possible
  • If this PR includes proposed changes for inclusion in the changelog, the title of this PR summarizes those changes and is ready for inclusion in the Changelog.
  • [] I have updated docstrings for function changes and docs in the 'docs' folder for user interface / behavior changes
  • This PR does not break any examples or I have updated them

(Optional) @mentions for Notifications:

@kevinlacaille kevinlacaille changed the base branch from maint-2.0 to main July 12, 2023 20:45
@kevinlacaille
Copy link
Contributor

kevinlacaille commented Jul 12, 2023

Great work, @JSee98!

  1. I changed the PR to merge to main. Could you please resolve the minor merge conflicts we're running into? If you need a had I'd happy to help you with that.
  2. It looks like mypy's type checking is failing. See if you can resolve this on your end. Try running mypy --ignore-missing planet in your terminal this branch. Typing can be a bit tricky to resolve, so please feel free to reach out if you need help with this.
  3. Below is a summary of what I'm seeing:

Previously:

planet.order_request.delivery("zip") 
#returns {'archive_type': 'zip'}

planet.order_request.delivery("zip", archive_filename="test.zip")
#returns {'archive_type': 'zip', 'archive_filename': 'test.zip'}

Now:

planet.order_request.delivery("zip") 
#returns {'archive_type': 'zip', 'single_archive': True}

planet.order_request.delivery("zip", archive_filename="test.zip")
#returns {'archive_type': 'zip', 'archive_filename': 'test.zip'}

This satisfies the desired logic where the delivery object should contain either:

    "archive_type": "zip",
    "single_archive": true

or

    "archive_type": "zip",
    "archive_filename": "{{name}}.zip"

Now, I just need to verify that an order actually delivers to a bucket (e.g., GCP bucket) using both of those methods.

Update: The data delivers as expected. See this comment for more detail: #989 (comment)

.gitignore Outdated
@@ -24,6 +24,6 @@ coverage.xml

# Editors
.vscode/

.idea/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what this is, but it isn't needed for our project. Could you please remove it from the file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this was just for Pycharm IDE. So if you use vscode as IDE the settings for the same are added under .vscode. Similarly, for IntelliJ products (Pycharm here), we get .idea to store the IDE settings. This shouldn't be a breaking change, however, I'll still remove it for the timebeing

Copy link
Contributor

@kevinlacaille kevinlacaille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, @JSee98!

  1. I just had 1 small change request for your test.
  2. There is a typing issue mypy is running into. See if you can resolve that locally, if not I'm happy to help.
  3. There appears to be a small merge conflict since I changed the base branch. Would you also be able to resolve that, please?

I'm going to test this out by delivering to a cloud bucket with this logic tomorrow. I recall having success delivering zipped files using single_archive=True and archive_filename=name.zip, but I'm hoping I was right in the ticket description I was correct that you could use either of those keywords. I'll report back tomorrow!

tests/unit/test_order_request.py Outdated Show resolved Hide resolved
@JSee98 JSee98 force-pushed the zipped-order-delivery-963 branch from 4b21e3a to a5dfa0e Compare July 13, 2023 06:34
@JSee98
Copy link
Contributor Author

JSee98 commented Jul 13, 2023

Also, nox isn't failing locally anymore, consequently mypy --ignore-missing planet command is also running fine locally. This must be due to the rebase with the main branch.

@kevinlacaille
Copy link
Contributor

Created an order where single_archive=True and archive_filename was unassigned: https://orders-ui.prod.planet-labs.com/orders/5ab30bb4-613b-4315-b44e-6a1bb60b556d

The delivery config contained:

{
    "google_cloud_storage": {
        "bucket": "devrel-notebooks",
        "credentials": "<REDACTED>"
    },
    "archive_type": "zip",
    "single_archive": true
}

Result: Order delivered as a "Single Zip", containing: manifest.json and output.zip, as one would expect.

Similarly, I created an order assigning archive_filename="test_filename.zip" and single_archive was unassigned: https://orders-ui.prod.planet-labs.com/orders/9a99249a-3db8-470b-8756-e5423a563d30

The delivery config contained:

{
    "google_cloud_storage": {
        "bucket": "devrel-notebooks",
        "credentials": "<REDACTED>"
    },
    "archive_type": "zip",
    "archive_filename": "test_filename.zip"
}

Result: Order delivered as a "Per Bundle Zip", containing: manifest.json and test_filename.zip, as one would expect.

@kevinlacaille kevinlacaille changed the title zipped-order-delivery-963: Updated delivery function in orders Updated order delivery config for zipped orders Jul 13, 2023
Copy link
Contributor

@sgillies sgillies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JSee98 thanks for the contribution! Can you and @kevinlacaille consider my suggestion?

@@ -199,6 +199,10 @@ def delivery(archive_type: Optional[str] = None,
if archive_type:
archive_type = specs.validate_archive_type(archive_type)

# for missing archive options set single_archive to false
if archive_filename is None and single_archive is False:
single_archive = True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JSee98 @kevinlacaille I have a different perspective on the problem. I don't think we want to change the single/multi flag, and thus the shape of the delivered artifacts, based on other inputs. Instead I think we want to make sure that the archive_filename defaults to some value if it is not specified. After a close read of https://developers.planet.com/apis/orders/delivery/#zipping-results, I believe we want something like this:

if archive_type:
    archive_filename = archive_filename or "{{name}}_{{order_id}}.zip"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to confirm @sgillies, your suggestion is if both the variables are undefined i.e. archive_filename and single_archive we should give a default value to archive_filename?
If you and @kevinlacaille agree on this suggested path I can make the required changes.

Copy link
Contributor

@sgillies sgillies Jul 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JSee98 we should provide a default for archive_filename whenever archive_type evaluates to True, no matter the value of single_archive. @kevinlacaille (traveling at the moment, so I'm speaking for him 😆) asked the API team and found that the need to specify an archive filename is a known Orders API bug and that this Python project is the right place to work around it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure guys, will make the updates and raise the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sgillies I've made the updates

@JSee98 JSee98 requested a review from sgillies July 16, 2023 13:14
Copy link
Contributor

@sgillies sgillies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JSee98 !

@sgillies sgillies merged commit 0aaf6c5 into planetlabs:main Jul 17, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to receive zipped deliveries to GCP
3 participants