Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get correct next offset for compacted topics #577

Merged
merged 8 commits into from
Dec 12, 2019
Merged

Get correct next offset for compacted topics #577

merged 8 commits into from
Dec 12, 2019

Commits on Dec 4, 2019

  1. Use Kafka 2.3 as default in testing

    @Nevon
    Nevon committed Dec 4, 2019
    Configuration menu
    Copy the full SHA
    c86d834 View commit details
    Browse the repository at this point in the history

  2. Resolve current offset when batch consists of only messages from befo… 

    …re the requested offset

Apparently this can happen when you get compacted topics, which can
cause you to get stuck, because there's no way to resolve any offsets.
What we do now in that case is to resolve the fetched offset, which will
advance by one.

We can't just use the highwatermark because you can get a batch
consisting only of these "old" compacted records while still having lots
of valid messages afterwards (so you don't want to skip to the end of
the partition).

Hopefully fixes #562
    @Nevon
    Nevon committed Dec 4, 2019
    Configuration menu
    Copy the full SHA
    912fe95 View commit details
    Browse the repository at this point in the history

  3. Update tests to work with Kafka 2.3

    @Nevon
    Nevon committed Dec 4, 2019
    Configuration menu
    Copy the full SHA
    7c2f69c View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2019

  1. Return fetched offset when empty due to compaction

    @Nevon
    Nevon committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    b579f66 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into get-correct-next-offset-for-compacted-topics

    @Nevon
    Nevon authored Dec 5, 2019
    Configuration menu
    Copy the full SHA
    f9fa09e View commit details
    Browse the repository at this point in the history

  3. Update pipeline to run with Kafka 2.3

    @Nevon
    Nevon committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    21b8264 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'get-correct-next-offset-for-compacted-topics' of github… 

    ….com:tulios/kafkajs into get-correct-next-offset-for-compacted-topics
    @Nevon
    Nevon committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    8a85884 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2019

  1. Merge branch 'master' into get-correct-next-offset-for-compacted-topics

    @tulios
    tulios authored Dec 9, 2019
    Configuration menu
    Copy the full SHA
    b3d9de4 View commit details
    Browse the repository at this point in the history