Skip to content

Default to THROUGHPUT_BASED autoscaling for classic templates #1403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
damccorm merged 6 commits into from
Apr 5, 2024
Merged

Default to THROUGHPUT_BASED autoscaling for classic templates #1403

damccorm merged 6 commits into from
Apr 5, 2024

Conversation

@damccorm
Copy link
Contributor

@damccorm damccorm commented Mar 27, 2024 •
edited
Loading

Before migrating the release process to GitHub, most streaming classic templates had autoscaling enabled by default. This functionality was lost as part of the move over, and this seeks to restore that default.

This has caused user pain because there's not an easy way to enable autoscaling for classic templates if its not on by default.

Note that I chose 10 workers as the default to avoid overwhelming users projects with high scaling (this happened to our templates project with integration tests when I tried 100) and because it seems like a reasonable default. It will cause streaming engine jobs without a default set to change their behavior (currently they default to 100 max workers)

@codecov
Copy link

codecov bot commented Mar 27, 2024 •
edited
Loading

Codecov Report

Merging #1403 (26d7c7c) into main (f014d34) will decrease coverage by 0.01%.
Report is 3 commits behind head on main.
The diff coverage is n/a.

❗ Current head 26d7c7c differs from pull request most recent head 0e186d9. Consider uploading reports for the commit 0e186d9 to get more accurate results

Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #1403      +/-   ##
============================================
- Coverage     38.93%   38.92%   -0.01%     
+ Complexity     2760     2759       -1     
============================================
  Files           743      743              
  Lines         41948    41948              
  Branches       4516     4516              
============================================
- Hits          16332    16329       -3     
- Misses        24127    24130       +3     
  Partials       1489     1489
Components Coverage Δ
spanner-templates 52.16% <ø> (ø)
spanner-import-export 65.22% <ø> (ø)
spanner-live-forward-migration 56.23% <ø> (ø)
spanner-live-reverse-replication 37.53% <ø> (ø)
spanner-bulk-migration 58.12% <ø> (ø)

see 1 file with indirect coverage changes

@damccorm damccorm marked this pull request as ready for review March 30, 2024 14:46
@damccorm damccorm requested a review from an2x March 30, 2024 14:46
an2x
an2x previously approved these changes Apr 4, 2024
View reviewed changes
@an2x an2x added the Google LGTM Approval of a pull request to be merged into the repository label Apr 4, 2024
liferoad
liferoad previously approved these changes Apr 4, 2024
View reviewed changes
@damccorm damccorm force-pushed the users/damccorm/classic-autoscaling branch from e7c6ed9 to 793f173 Compare April 5, 2024 12:55
@damccorm damccorm dismissed stale reviews from an2x and liferoad via 928dc6b April 5, 2024 14:47
@pull-request-size pull-request-size bot added size/S and removed size/XS labels Apr 5, 2024
@pull-request-size pull-request-size bot added size/XS and removed size/S labels Apr 5, 2024
@damccorm damccorm merged commit 980e7ab into main Apr 5, 2024
@damccorm damccorm deleted the users/damccorm/classic-autoscaling branch April 5, 2024 17:53
damccorm added a commit that referenced this pull request Apr 11, 2024
@damccorm
Merge pull request #1403 from GoogleCloudPlatform:users/damccorm/clas…
edf9314 
…sic-autoscaling

PiperOrigin-RevId: 622168528
damccorm added a commit that referenced this pull request Apr 11, 2024
@damccorm
Merge pull request #1403 from GoogleCloudPlatform:users/damccorm/clas…
4d6990a 
…sic-autoscaling

PiperOrigin-RevId: 622188398
liferoad pushed a commit to aksharauke/DataflowTemplates that referenced this pull request Apr 18, 2024
@aksharauke @liferoad
removed deprecated reverse replicaiton templates
2c59830 
Get consistent java-pr signal on main

Default to THROUGHPUT_BASED autoscaling for classic templates (GoogleCloudPlatform#1403)

* single commit off of master

* No-op comment change to kick copybara

* Revert "No-op comment change to kick copybara"

This reverts commit 928dc6b.

* Different no-op to kick copybara and avoid conflicts

* One more try to get copybara to merge

Support FLOAT32 type in v1 templates of Spanner. (GoogleCloudPlatform#1376)

Squashed commit of the following:

commit 1e21615
Merge: 5764ef8 27e7486
Author: Aravind Pedapudi <aravindp1510@gmail.com>
Date:   Thu Apr 4 21:35:22 2024 +0530

    Merge branch 'main' into float32

commit 5764ef8
Author: Aravind <aravindp1510@gmail.com>
Date:   Mon Apr 1 17:40:34 2024 +0530

    Support FLOAT32 type in v1 templates of Spanner.

    Squashed commit of the following:

    commit a678a75
    Merge: f9fe874 c9f1c66
    Author: Aravind Pedapudi <aravindp1510@gmail.com>
    Date:   Mon Apr 1 16:08:20 2024 +0530

        Merge branch 'main' into float32

    commit f9fe874
    Merge: e28335c 7449ae6
    Author: Aravind Pedapudi <aravindp1510@gmail.com>
    Date:   Thu Mar 28 10:50:24 2024 +0530

        Merge branch 'main' into float32

    commit e28335c
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Wed Mar 27 16:33:56 2024 +0530

        Fix integration test failures for TextImportPipelineIT

    commit 8480654
    Merge: c6364b7 afd1dc2
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Wed Mar 27 16:25:57 2024 +0530

        Merge branch 'float32' of github.com:arawind/DataflowTemplates into float32

    commit c6364b7
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Wed Mar 27 16:22:27 2024 +0530

        Add avro integration tests for FLOAT32 type in Spanner.

        This fixes the ExportPipelineIT and adds support for the
        ImportPipelineIT.

    commit afd1dc2
    Merge: eca6acf 4536653
    Author: Aravind Pedapudi <aravindp1510@gmail.com>
    Date:   Wed Mar 27 11:37:24 2024 +0530

        Merge branch 'main' into float32

    commit eca6acf
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Fri Mar 22 17:12:35 2024 +0530

        Address review comments

        Adds new tests in ExportPipelineIT and also removes the INT32 -> FLOAT32 parsing as we don't see any use for it.

    commit c1cd281
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Thu Mar 21 18:15:05 2024 +0530

        Fix Beam to handle float32 arrays and pg schema.

        This was missed in the earlier commits

    commit 5e936f4
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Thu Mar 21 13:32:01 2024 +0530

        Address review comments

    commit f4ab908
    Merge: a6aba30 1df7e0a
    Author: Aravind Pedapudi <aravindp1510@gmail.com>
    Date:   Thu Mar 21 12:56:09 2024 +0530

        Merge branch 'GoogleCloudPlatform:main' into float32

    commit a6aba30
    Merge: 66c488c f44f7ce
    Author: Aravind Pedapudi <aravindp1510@gmail.com>
    Date:   Wed Mar 20 16:21:11 2024 +0530

        Merge branch 'main' into float32

    commit 66c488c
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Wed Mar 20 15:49:31 2024 +0530

        Fix gax dependency failures in spanner integration tests.

        We were seeing this error in the spanner integration tests: `class com.google.cloud.spanner.v1.stub.SpannerStubSettings overrides final method com.google.api.gax.rpc.StubSettings.getEndpoint()Ljava/lang/String;`, which were triggered by the google-cloud-spanner BOM update to 6.61.
        We're fixing these errors by setting the gax version to the one used by Cloud Spanner client libs. This should be removed once Beam is updated to the latest version of gax (which may happen when Beam's dependency on Spanner is updated).

    commit 90c1fbc
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Tue Mar 19 19:17:03 2024 +0530

        Support FLOAT32 type in v1 spanner templates

    commit 51f75f0
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Tue Mar 19 19:10:06 2024 +0530

        Modify the local copy of Beam to support FLOAT32 type.

    commit f85fd00
    Author: Aravind <aravindp1510@gmail.com>
    Date:   Tue Mar 19 18:53:17 2024 +0530

        Update the version of google-cloud-spanner-bom to 6.61.0

Upgrade Beam version to 2.55.1

Specify venv

single commit off of master

No-op comment change to kick copybara

Revert "No-op comment change to kick copybara"

This reverts commit 928dc6b.

Different no-op to kick copybara and avoid conflicts

PR GoogleCloudPlatform#1376: Support FLOAT32 type in v1 templates of Spanner

Please approve this CL. It will be submitted automatically, and its GitHub pull request will be marked as merged.

Imported from GitHub PR GoogleCloudPlatform#1376

GoogleSQL:
* [Export](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-21_05_01_22-9618753751624897563;graphView=0?project=span-cloud-testing&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)))
* [Import](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-21_05_38_00-8680174928377892705;graphView=0?project=span-cloud-testing&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)))

PG:
* [Export](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-21_05_47_07-2771163877050158940;graphView=0?project=span-cloud-testing&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)))
* [Import](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-21_05_58_11-1643174307354839989;graphView=0?project=span-cloud-testing&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)))

GoogleSQL:
* [Export](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-22_00_17_56-5509502875882896332?e=13802955&mods=monitoring_api_prod&project=span-cloud-testing)
* [Import](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-22_02_50_39-14326646282971458157?e=13802955&mods=monitoring_api_prod&project=span-cloud-testing)

PG:
* [Export](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-22_03_17_13-5127588554893435640;graphView=0?project=span-cloud-testing&e=13802955&mods=monitoring_api_prod&pageState=(%22dfTime%22:(%22l%22:%22dfJobMaxTime%22)))
* [Import](https://console.cloud.google.com/dataflow/jobs/us-central1/2024-03-22_03_09_55-9477941764979866789?e=13802955&mods=monitoring_api_prod&project=span-cloud-testing)

Copybara import of the project:

  - 357481bb0fdfba9c91fdde1c24ae4bbcaf5d2bf4 Support FLOAT32 type in v1 templates of Spanner. by Aravind <aravindp1510@gmail.com>

COPYBARA_INTEGRATE_REVIEW=GoogleCloudPlatform#1376 from arawind:float32 357481bb0fdfba9c91fdde1c24ae4bbcaf5d2bf4
PiperOrigin-RevId: 622232578

String change to match external

PiperOrigin-RevId: 622247573

Add schemaMapperInterface and implementations

Add namespace to params

spotless

Add user-agent string to Elasticsearch templates

Rename templateName to userAgent

Fix style

Capitalize header name

Add unit tests

Fix style

Bugfix for UI issue removing flex templates from drop down menu

Use beam version defined in the pom.xml to make sure we stay on the most current version. Also add a comment regarding the hashed file location for context.

Implement enum based experiment value provider

disable YamlTemplateIT's until GoogleCloudPlatform#1405 is merged

Signed-off-by: Jeffrey Kinard <jeff@thekinards.com>

Add smoke tests for the no UDF case for Xlang templates, abstract conversion functions, fix bug with flexContainerName.

A small simplification in Bulk Reader tests.

Allow manual triggering of PR workflow

Add python version to setup-env actions

Add parentName and parentTriggerValues attributes

Test that new attributes are present

Remove redundant line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liferoad liferoad liferoad left review comments

@an2x an2x an2x left review comments

Assignees

No one assigned

Labels

Google LGTM Approval of a pull request to be merged into the repository size/XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@damccorm @liferoad @an2x