Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add WorkflowTemplate run-fv3gfs for segmented long runs #576

Merged
merged 65 commits into from
Aug 26, 2020
Merged

Add WorkflowTemplate run-fv3gfs for segmented long runs #576

merged 65 commits into from
Aug 26, 2020

Commits on Aug 14, 2020

  1. Add initial workflow for segmented fv3gfs run

    Oliver Watt-Meyer committed Aug 14, 2020
    Configuration menu
    Copy the full SHA
    7f925de View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2020

  1. Initial implememtation of shift_chunks and append_run

    Oliver Watt-Meyer committed Aug 15, 2020
    Configuration menu
    Copy the full SHA
    a2adeea View commit details
    Browse the repository at this point in the history

  2. Add temporary config/scripts for testing workflow

    Oliver Watt-Meyer committed Aug 15, 2020
    Configuration menu
    Copy the full SHA
    1f0ab96 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2020

  1. Ensure _shift_store works with dim with coord

    Oliver Watt-Meyer committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    3f62ade View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into feature/segmented-fv3gfs-runs

    Oliver Watt-Meyer committed Aug 18, 2020
    Configuration menu
    Copy the full SHA
    63c3af8 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2020

  1. Fix append_run main. Uses output from post_process

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    bd499aa View commit details
    Browse the repository at this point in the history

  2. Use initial timestamp as segment label

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    03b05d1 View commit details
    Browse the repository at this point in the history

  3. Add append_run.py to post_process Docker image

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    16fae7d View commit details
    Browse the repository at this point in the history

  4. Remove unused shutil import

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    866ed8d View commit details
    Browse the repository at this point in the history

  5. Change argo workflow to use append_run to do uploads

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    72f8037 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into feature/segmented-fv3gfs-runs

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    3ffe137 View commit details
    Browse the repository at this point in the history

  7. Make sure append_run.py runs as python

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    e230d1d View commit details
    Browse the repository at this point in the history

  8. Install fsspec and gcsfs in post_process_run docker image

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    7b1f629 View commit details
    Browse the repository at this point in the history

  9. Skip symlinks for gsutil upload step

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    53bf2ae View commit details
    Browse the repository at this point in the history

  10. Allow post process script to output to local dir

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    eaba512 View commit details
    Browse the repository at this point in the history

  11. Post-process and then append run for upload

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    a7b9633 View commit details
    Browse the repository at this point in the history

  12. Update test script to use latest post-process image

    Oliver Watt-Meyer committed Aug 19, 2020
    Configuration menu
    Copy the full SHA
    12afa9e View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2020

  1. Convert time units for any time-like variables

    Oliver Watt-Meyer committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    673fee3 View commit details
    Browse the repository at this point in the history

  2. Remove unused import

    Oliver Watt-Meyer committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    1754bc1 View commit details
    Browse the repository at this point in the history

  3. Clear /mnt/data/rundir and /tmp/processed_rundir after each step

    Oliver Watt-Meyer committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    6326c85 View commit details
    Browse the repository at this point in the history

  4. Add more logging statements to append_run main

    Oliver Watt-Meyer committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    e860c96 View commit details
    Browse the repository at this point in the history

  5. Save restarts to shared volume

    Oliver Watt-Meyer committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    00269c6 View commit details
    Browse the repository at this point in the history

  6. Do append run work within tempdir

    Oliver Watt-Meyer committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    7f7ec71 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2020

  1. Add more unit tests for append_run, fix order of renaming

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    fb3e2ca View commit details
    Browse the repository at this point in the history

  2. Use run-fv3gfs template within nudging template

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    a7e4be5 View commit details
    Browse the repository at this point in the history

  3. Delete unused imports

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    4f8286c View commit details
    Browse the repository at this point in the history

  4. Change run-fv3gfs to a workflow template

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    196f5a4 View commit details
    Browse the repository at this point in the history

  5. Remove temporary test configs/scripts

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    6343b4d View commit details
    Browse the repository at this point in the history

  6. Fix e2e integration test

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    23976b0 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into feature/segmented-fv3gfs-runs

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    bf82aaa View commit details
    Browse the repository at this point in the history

  8. Add README section for run-fv3gfs

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    588918b View commit details
    Browse the repository at this point in the history

  9. Rename output-data volume claim to workdir

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    2e68d93 View commit details
    Browse the repository at this point in the history

  10. Add persistent-volume arg to run-fv3gfs template

    Oliver Watt-Meyer committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    8febfd9 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'master' into feature/segmented-fv3gfs-runs

    Oliver Watt-Meyer authored Aug 21, 2020
    Configuration menu
    Copy the full SHA
    925407b View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2020

  1. Try (and fail) to make volume mounts better

    Oliver Watt-Meyer committed Aug 22, 2020
    Configuration menu
    Copy the full SHA
    1331d72 View commit details
    Browse the repository at this point in the history

  2. Hard code volume names

    Oliver Watt-Meyer committed Aug 22, 2020
    Configuration menu
    Copy the full SHA
    752320b View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2020

  1. Post-process with user-provided chunks

    Oliver Watt-Meyer committed Aug 24, 2020
    Configuration menu
    Copy the full SHA
    1fce45b View commit details
    Browse the repository at this point in the history

  2. Make chunks argument optional. Default to empty dict

    Oliver Watt-Meyer committed Aug 24, 2020
    Configuration menu
    Copy the full SHA
    abac080 View commit details
    Browse the repository at this point in the history

  3. Echo result of argo get if timed out or success ambiguous

    Oliver Watt-Meyer committed Aug 24, 2020
    Configuration menu
    Copy the full SHA
    7b0c996 View commit details
    Browse the repository at this point in the history

  4. Specify chunks default in template inputs.parameters

    Oliver Watt-Meyer committed Aug 24, 2020
    Configuration menu
    Copy the full SHA
    fa378c4 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2020

  1. Use recursion for loop so that subsequent steps fail fast

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    2d34601 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into feature/segmented-fv3gfs-runs

    Oliver Watt-Meyer authored Aug 25, 2020
    Configuration menu
    Copy the full SHA
    ff53a96 View commit details
    Browse the repository at this point in the history

  3. Generate working volume in run-fv3gfs workflow template

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    2c0f223 View commit details
    Browse the repository at this point in the history

  4. Revert "Generate working volume in run-fv3gfs workflow template" 

    This reverts commit 2c0f223.
    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    41d6ea2 View commit details
    Browse the repository at this point in the history

  5. Make volume claims in entrypoint workflow

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    048e124 View commit details
    Browse the repository at this point in the history

  6. Update README

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    90f708c View commit details
    Browse the repository at this point in the history

  7. Apply suggestions from code review 

    Co-authored-by: Noah D Brenowitz <nbren12@gmail.com>
Co-authored-by: Spencer Clark <spencerkclark@gmail.com>
    @nbren12 @spencerkclark
    3 people authored Aug 25, 2020
    Configuration menu
    Copy the full SHA
    28a590d View commit details
    Browse the repository at this point in the history

  8. Improve volume naming and docs

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    bae1d99 View commit details
    Browse the repository at this point in the history

  9. Pithier volume name

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    5d88fda View commit details
    Browse the repository at this point in the history

  10. Rename submission-count to segment-count

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    714ae94 View commit details
    Browse the repository at this point in the history

  11. Add some explanatory comments to run-fv3gfs workflow

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    7065ebb View commit details
    Browse the repository at this point in the history

  12. Mention gcp-secret-key volume

    Oliver Watt-Meyer committed Aug 25, 2020
    Configuration menu
    Copy the full SHA
    cf114cc View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2020

  1. Define new public func append_zarr_along_time

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    fcfd66b View commit details
    Browse the repository at this point in the history

  2. Remove trailing whitespace

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    664cff2 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into feature/segmented-fv3gfs-runs

    Oliver Watt-Meyer authored Aug 26, 2020
    Configuration menu
    Copy the full SHA
    323ab9a View commit details
    Browse the repository at this point in the history

  4. Rename append_run to append_segment

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    a900fe9 View commit details
    Browse the repository at this point in the history

  5. Move test_append_run to test_append

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    38f37c9 View commit details
    Browse the repository at this point in the history

  6. Delete copy_tree implementation in test_append

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    e626060 View commit details
    Browse the repository at this point in the history

  7. Remove unused shutil import

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    1f242c3 View commit details
    Browse the repository at this point in the history

  8. Fix wrong arg name

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    c15ce0d View commit details
    Browse the repository at this point in the history

  9. Actually fix

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    7e312de View commit details
    Browse the repository at this point in the history

  10. Use consolidated=True to open nudging data in e2e test

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    3d224d1 View commit details
    Browse the repository at this point in the history

  11. Format parameter table in README

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    8a7b4d0 View commit details
    Browse the repository at this point in the history

  12. NB rereview comments on append

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    f448e86 View commit details
    Browse the repository at this point in the history

  13. Echo output of argo terminate in e2e run_test.sh

    Oliver Watt-Meyer committed Aug 26, 2020
    Configuration menu
    Copy the full SHA
    6d7d335 View commit details
    Browse the repository at this point in the history