Skip to content

[SPARK-27642][SS] make v1 offset extends v2 offset #24538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-27642][SS] make v1 offset extends v2 offset #24538

wants to merge 1 commit into from

Conversation

cloud-fan
Copy link
Contributor

What changes were proposed in this pull request?

To move DS v2 to the catalyst module, we can't make v2 offset rely on v1 offset, as v1 offset is in sql/core.

How was this patch tested?

existing tests

@cloud-fan
Copy link
Contributor Author

cc @jose-torres @gatorsmile

@SparkQA
Copy link

SparkQA commented May 6, 2019

Test build #105157 has finished for PR 24538 at commit 47c49a3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class KafkaSourceOffset(partitionToOffsets: Map[TopicPartition, Long]) extends Offset
  • public abstract class Offset extends org.apache.spark.sql.sources.v2.reader.streaming.Offset
  • public abstract class Offset
  • case class LongOffset(offset: Long) extends Offset
  • case class OffsetSeq(offsets: Seq[Option[OffsetV2]], metadata: Option[OffsetSeqMetadata] = None)

gatorsmile
gatorsmile reviewed May 8, 2019
View reviewed changes
Copy link
Member

@gatorsmile gatorsmile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks! Merged to master.

@gatorsmile gatorsmile closed this in bae5baa May 8, 2019
mccheah pushed a commit to palantir/spark that referenced this pull request Jun 6, 2019
@cloud-fan @mccheah
[SPARK-27642][SS] make v1 offset extends v2 offset
12746c1 
## What changes were proposed in this pull request?

To move DS v2 to the catalyst module, we can't make v2 offset rely on v1 offset, as v1 offset is in sql/core.

## How was this patch tested?

existing tests

Closes apache#24538 from cloud-fan/offset.

Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gatorsmile gatorsmile gatorsmile left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cloud-fan @SparkQA @gatorsmile