Add Priortized CUDs attribution tool. #395
Conversation
tools/cuds-prioritized-attribution/test/test_commitment_intervals.py
Outdated
Show resolved
Hide resolved
|
|
||
|
|
||
| # Upload dependencies folder to DAG folder in Composer Bucket | ||
| resource "google_storage_bucket_object" "dag_init_upload" { |
There was a problem hiding this comment.
I've been told it is poor form to deploy software like this with terraform.
Would cloud build be a more appropriate tool staging all these files for this?
There was a problem hiding this comment.
Possibly, I used the same terraform recommendations from this tool, as recommended by morgante.
There was a problem hiding this comment.
Respectfully, disagree that this is a similar pattern to the other tool
In the deployment process for the other tool you zip up source trees.
Here you require a contributor to manage each source file (including pesky __init__.py's) as a separate GCS object which is much more error prone.
Having a deployment process that runs `gsutil -m rsync -r ${local_dags_dir} gs://${composer_bucket}/dags would be much preferable.
PTAL at this simple cloud build example we have for deploying dags and see if you can adapt it for your use in this asset.
Aside,
We're working on a more fully baked CI/CD process for composer (deck, source) for composer but it is not quite ready yet.
|
@kardiff18 |
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
googlebot
added
cla: no
|
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
googlebot
added
cla: yes
tools/cuds-prioritized-attribution/composer/dependencies/commitment_intervals.py
Outdated
Show resolved
Hide resolved
tools/cuds-prioritized-attribution/composer/dependencies/commitment_intervals.py
Outdated
Show resolved
Hide resolved
| schedule_interval=datetime.timedelta(days=1), | ||
| default_args=DEFAULT_DAG_ARGS) as dag: | ||
|
|
||
| def format_commitment_table(env_vars: Dict[str, Union[str, bool]]) -> None: |
… functions location
|
This is getting close:
|
|
Simplified GCS upload in both CLI & terraform to be one step, meaning that I added in a folder within the DAG rather than doing everything at the root. @arifkasim this might affect your tests - please rerun them. Also changed to jinja templating. I've tested the full workflow in both the CLI & terraform and verifies that it works and produces the output table. Once Arif can run through his test suite once to verify outputs match, then should be good from our side. @jaketf PTAL and let us know if there's anything else, thanks! |
|
@kardiff18 I made the changes to tests to be compatible with jinja templates. All tests work now. |
A tool that allows customers to prioritize a specific scope (e.g. project or folder) to attribute committed use discounts to first before letting any unconsumed discount float to other parts of an organization.
PSO Eng Bug Id = 135540892