[BEAM-5392] group by key non merging window optimalization by mareksimunek · Pull Request #49 · seznam/beam

mareksimunek · 2018-10-25T15:02:02Z

Follow this checklist to help us incorporate your contribution quickly and easily:

Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go	---	---	---	---	---	---
Java
Python	---			---	---	---

dmvk · 2018-10-26T13:22:33Z

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java

dmvk · 2018-10-26T13:24:23Z

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java

dmvk · 2018-10-26T13:30:50Z

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java

GroupByKeyIteratoror

dmvk · 2018-10-26T13:33:37Z

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java

…ynthetic data to Kafka topic

This prints a warning when checkpointing is disabled and pipelines are run with the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on checkpoint acknowledgement, e.g. PubSubIO.

…ython connector Write

…ogle-api-core 1.7.0

…of null constructor in ByteArrayShufflePosition

- MetricKey - MetricName - MetricQueryResults - MetricsContainerStepMap

…ommit java

… support for Java SDK

…e inputs

Generic types are de-facto @nullable so this annotation is redundant and probably only existed to make FindBugs happy.

…zed side inputs between Spark tasks

…n encoding Key columns.

Adds withKmsKey() methods to reads and writes. Affects tables created tables by: streaming writes, batched writes, query temporary tables, table copies (for very large batched writes).

[BEAM-5667] Use BeamDeprecationWarning in BQ client code.

[BEAM-6636] Remove --kmsKey usage

…y3 limitations. (apache#7773) * Restrict pyvcf dependency to Python 2 and link Jira issues tracking Py3 limitations. * Skip all VCF IO tests so that we don't need to install pyvcf on Python 3. * Don't import VCF on Python 2.

…get()" This reverts commit 72627eb.

…unner

…for kryo registration in Spark runner

[BEAM-5959] KMS support for BigQuery

…transforms (apache#7709) Multi-language support in DefaultJobBundleFactory

…f manual classpath

…r interval >= 25 DAY [BEAM-6632] Fixes integer overflow for interval >= 25 DAY

…iption

mareksimunek requested review from VaclavPlajt and dmvk October 25, 2018 15:02

dmvk reviewed Oct 26, 2018

View reviewed changes

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java Outdated

Copy link

dmvk Oct 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo??

dmvk reviewed Oct 26, 2018

View reviewed changes

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java Outdated

Copy link

dmvk Oct 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsigned

dmvk reviewed Oct 26, 2018

View reviewed changes

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java Outdated

Copy link

dmvk Oct 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GroupByKeyIteratoror

dmvk reviewed Oct 26, 2018

View reviewed changes

...src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java Outdated

Copy link

dmvk Oct 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

private

mareksimunek force-pushed the simunek/groupByKeyNonMergingWin/BEAM-5392 branch from 33ef415 to 41d4b8f Compare November 7, 2018 15:05

mareksimunek force-pushed the simunek/groupByKeyNonMergingWin/BEAM-5392 branch from ab8e5e9 to 997caf4 Compare December 19, 2018 16:58

mareksimunek force-pushed the simunek/groupByKeyNonMergingWin/BEAM-5392 branch 2 times, most recently from ceaee54 to b95a484 Compare January 22, 2019 09:56

mareksimunek changed the base branch from beamMaster to master January 22, 2019 10:57

mareksimunek changed the base branch from master to beamMaster January 22, 2019 10:57

mareksimunek force-pushed the simunek/groupByKeyNonMergingWin/BEAM-5392 branch 6 times, most recently from d66cbeb to 5189810 Compare January 25, 2019 14:05

lgajowy and others added 12 commits January 30, 2019 16:02

Merge pull request apache#7612: [BEAM-6207] Added option to publish s…

d1ac67c

…ynthetic data to Kafka topic

Sync dependencies in base_image_requirements.txt with setup.py

ea2c5d1

[BEAM-4520] Warn when UnboundedSources are used without checkpointing

fe27329

This prints a warning when checkpointing is disabled and pipelines are run with the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on checkpoint acknowledgement, e.g. PubSubIO.

[BEAM-4520] Add note on checkpointing of UnboundedSources

8803821

This prints a warning when checkpointing is disabled and pipelines are run with the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on checkpoint acknowledgement, e.g. PubSubIO.

Merge pull request apache#7367: [BEAM-3342] Create a Cloud Bigtable P…

054d42c

…ython connector Write

Update minimal version of futures due to transitive requirement in go…

4a74b55

…ogle-api-core 1.7.0

Merge pull request apache#7667: [BEAM-6545] Remove useless allowance …

bb49247

…of null constructor in ByteArrayShufflePosition

add toString()s to Metrics classes

140ea54

- MetricKey - MetricName - MetricQueryResults - MetricsContainerStepMap

add MetricResult.result wrapper around committed || attempted

150a7a6

fix MetricQueryResults comments

8976c9e

[BEAM-5091] Wordcount integration test on DataflowRunner in Python 3

e850105

add post build action to record errorpron/checkstyle/findbugs on prec…

dcd9ff0

…ommit java

iemejia and others added 7 commits February 8, 2019 13:41

[BEAM-6631] Update commons-compress to version 1.18

25317b5

Merge pull request apache#7416: [BEAM-6365] Add ZStandard compression…

c96b096

… support for Java SDK

[BEAM-5987] Cache and share materialized side inputs between Spark tasks

4d89396

[BEAM-5987] Use Guava's Cache to control eviction of materialized sid…

a9117f9

…e inputs

[BEAM-5987] Remove @nullable annotation from SideInputReader.get()

72627eb

Generic types are de-facto @nullable so this annotation is redundant and probably only existed to make FindBugs happy.

Merge pull request apache#7091: [BEAM-5987] Cache and share materiali…

f1bb53a

…zed side inputs between Spark tasks

BEAM-6632: fix integer overflow for interval >= 25

0fef9e1

mareksimunek force-pushed the simunek/groupByKeyNonMergingWin/BEAM-5392 branch from 835132f to 5a7e8fa Compare February 8, 2019 17:05

chamikaramj and others added 20 commits February 8, 2019 10:34

Merge pull request apache#7747: [BEAM-4359] Check for null values whe…

1512551

…n encoding Key columns.

Make pydocs happy.

af06ac5

[BEAM-5959] KMS support for BigQuery

d4e05a1

Adds withKmsKey() methods to reads and writes. Affects tables created tables by: streaming writes, batched writes, query temporary tables, table copies (for very large batched writes).

[BEAM-6636] Remove --kmsKey usage

6b888e9

Merge pull request apache#7765 from tvalentyn/fix_warnings

ecc8538

[BEAM-5667] Use BeamDeprecationWarning in BQ client code.

Merge pull request apache#7790 from udim/cmek-bq-gradle

2f9f61f

[BEAM-6636] Remove --kmsKey usage

Revert "[BEAM-5987] Remove @nullable annotation from SideInputReader.…

7bc0a8c

…get()" This reverts commit 72627eb.

[BEAM-6426] Use vendored guava class for kryo registration in Spark r…

652166b

…unner

Merge pull request apache#7579: Update python-pipeline-dependencies.md

c86b677

Merge pull request apache#7789: [BEAM-6426] Use vendored guava class …

b8880ef

…for kryo registration in Spark runner

Merge pull request apache#7744 from udim/cmek-bq

49403c0

[BEAM-5959] KMS support for BigQuery

[BEAM-4102] use plugins DSL instead of manual classpath

531cdf5

[BEAM-6488] Portable Flink runner support for running cross-language …

c50e3fc

…transforms (apache#7709) Multi-language support in DefaultJobBundleFactory

BEAM-6632: Use UTC timezone in test

ca7bc1d

BEAM-6632: added todo comment related to CALCITE-2837

708434a

Merge pull request apache#7781: [BEAM-4102] use plugins DSL instead o…

7940259

…f manual classpath

Merge pull request apache#7785: [BEAM-6632] Fixes integer overflow fo…

98d7f0d

…r interval >= 25 DAY [BEAM-6632] Fixes integer overflow for interval >= 25 DAY

[BEAM-5392] GroupByKey optimized for non-merging windows

4a1b983

[BEAM-5392] small CR corrections

023f31d

mareksimunek force-pushed the simunek/groupByKeyNonMergingWin/BEAM-5392 branch from 5a7e8fa to 023f31d Compare February 11, 2019 09:41

[BEAM-5392] GroupNonMergingWindowsFunctions.groupByKeyAndWindow descr…

6283f3d

…iption

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-5392] group by key non merging window optimalization#49

[BEAM-5392] group by key non merging window optimalization#49
mareksimunek wants to merge 451 commits intobeamMasterfrom
simunek/groupByKeyNonMergingWin/BEAM-5392

mareksimunek commented Oct 25, 2018

Uh oh!

dmvk Oct 26, 2018

Uh oh!

dmvk Oct 26, 2018

Uh oh!

dmvk Oct 26, 2018

Uh oh!

dmvk Oct 26, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

mareksimunek commented Oct 25, 2018

Post-Commit Tests Status (on master branch)

Uh oh!

dmvk Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

dmvk Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

dmvk Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

dmvk Oct 26, 2018

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants