[test][broker] PIP-475: end-to-end migration tests + transition fixes#25878
Merged
Conversation
Adds V5MigrationEndToEndTest covering the full operator timeline against a live broker, and fixes three real bugs the test surfaced that prevented a connected V5 client from transitioning across the migration boundary. Tests (V5MigrationEndToEndTest): * produce-through-migration: a V5 producer publishes via the synthetic layout, the topic is migrated while only that (marked) V5 producer is attached, the producer transparently follows the layout change to the real DAG, and a V5 queue consumer drains every pre- and post-migration message. * v4 lockout: after migration the old topic is terminated, so a legacy v4 producer can no longer write to it. Fixes: * DagWatchSession: compute segment:// URIs from the canonical topic:// name, not the session's raw input. A session opened with a persistent:// name (synthetic layout) previously threw "Parent topic must have domain 'topic'" inside buildResponse when the topic was migrated, so the real DAG was never pushed and connected clients never transitioned. * ScalableTopics migration pre-check: inspect per-partition stats instead of the aggregate. Aggregated partitioned stats merge publishers by name into fresh stat objects that drop per-connection metadata, which hid the V5-managed marker and made every V5 connection look like a legacy v4 one. * ScalableTopicProducer: the send-retry now also covers per-segment producer *creation* (not just send()), since a migration terminates the old partition between routing and creation; detect the "segment gone" condition by unwrapping causes (type or message) and give the DAG-watch layout update a larger budget to arrive.
…tion Adds TestScalableTopicMigration (tests/integration), which runs `pulsar-admin scalable-topics migrate` against a real multi-broker dockerized cluster: seed a partitioned regular topic via a v4 client, migrate it via the admin CLI, assert the topic is now scalable (get-metadata returns the segment DAG), and assert a legacy v4 producer is locked out afterward (the old partitions are terminated). Registered in the pulsar-messaging integration suite. Complements the in-process V5MigrationEndToEndTest (which covers the V5-client transparent transition); this validates the command, CLI wiring, and termination in a real deployment with a real metadata store, BookKeeper, and cross-broker bundle ownership.
Split the combined instanceof check into two single-line ifs so neither line wraps an operator (OperatorWrap). No behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final implementation PR for PIP-475: Regular-to-Scalable Topic Migration. Builds on the synthetic-layout lookup (#25822), the V5 SDK support (#25850), and the migration command (#25875). Adds an end-to-end test of the full operator timeline against a live broker, and fixes three real bugs the test surfaced that prevented a connected V5 client from transitioning across the migration boundary.
Tests (
V5MigrationEndToEndTest)--force; the producer transparently follows the layout-change push to the real DAG and range-routes new messages to the active children; a V5 queue consumer then drains every pre- and post-migration message.Fixes surfaced by the E2E test
DagWatchSession(the transition gap): computesegment://URIs from the canonicaltopic://name, not the session's raw input. A session opened with apersistent://name (synthetic layout) previously threw "Parent topic must have domain 'topic'" insidebuildResponseonce the topic was migrated, so the real DAG was never pushed and connected clients never transitioned.ScalableTopicProducer: the send-retry now also covers per-segment producer creation (not justsend()), since a migration terminates the old partition between routing and creation; detect the "segment gone" condition by unwrapping causes (type or message) and give the DAG-watch layout update a larger budget to arrive.Test plan
V5MigrationEndToEndTest(2): produce-through-migration, v4 lockout.ScalableTopicMigrationTest7/7,DagWatchSessionTest18/18,ScalableTopicControllerTest35/35,V5RegularTopicInteropTest4/4, fullpulsar-client-v5unit suite.