Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Ticdc output data to pulsar #9413

Closed
yumchina opened this issue Jul 20, 2023 · 2 comments
Closed

Support Ticdc output data to pulsar #9413

yumchina opened this issue Jul 20, 2023 · 2 comments
Assignees
@asddongmen @yumchina
Labels
area/ticdc Issues or PRs related to TiCDC. type/feature Issues about a new feature

Comments

@yumchina
Copy link
Contributor

yumchina commented Jul 20, 2023 •
edited
Loading

Is your feature request related to a problem?

Ticdc can output data to pulsar

Describe the feature you'd like

Ticdc output data to pulsar like kafka.
1 Support some config with a pulsar client.
2 Ticdc output DDL/DML messages to pulsar with Canal/Canal-Json/Maxwell protocol.
I will make a pulsar sink to output data to pulsar. All data formats will be similar to Kafka's, but because the client is different, the details will be slightly different.
This new function will provide ticdc with the ability to output data to pulsar, allowing users to reuse existing pulsar clusters.

Describe alternatives you've considered

No

Teachability, Documentation, Adoption, Migration Strategy

ticdc-pulsar-sink

@yumchina yumchina added the type/feature Issues about a new feature label Jul 20, 2023
This was referenced Jul 20, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jul 21, 2023
@yumchina
ticdc 2 pulsar : add pulsar config from sink-uri & pulsar errors (#9393)
db5e513 
ref #9413
@yumchina yumchina mentioned this issue Jul 26, 2023
Merged
@nongfushanquan
Copy link
Contributor

/area ticdc

@ti-chi-bot ti-chi-bot bot added the area/ticdc Issues or PRs related to TiCDC. label Jul 26, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jul 27, 2023
@yumchina
sink (pulsar): add Expression Pulsar Validate for topic name and unit…
1dc0555 
… test . (#9401)

ref #9413
ti-chi-bot bot pushed a commit that referenced this issue Aug 7, 2023
@yumchina
ticdc(pulsar): write DDL events to pulsar (#9433)
e616696 
ref #9413
This was referenced Aug 15, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 24, 2023
@yumchina
ticdc(sink) move pulsar sink-uri configs to sink of replica config (#…
c903b87 
…9578)

ref #9413
ti-chi-bot bot pushed a commit that referenced this issue Aug 25, 2023
@yumchina
ticdc(sink) Write DML events to pulsar (#9622)
c703296 
ref #9413
@yumchina yumchina mentioned this issue Aug 28, 2023
Closed
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Aug 29, 2023
@yumchina @3AceShowHand
ticdc(pulsar): write DDL events to pulsar (pingcap#9433)
a199d59 
ref pingcap#9413
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Aug 29, 2023
@yumchina @3AceShowHand
ticdc(sink) move pulsar sink-uri configs to sink of replica config (p…
24d8ded 
…ingcap#9578)

ref pingcap#9413
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Aug 29, 2023
@yumchina @3AceShowHand
ticdc(sink) Write DML events to pulsar (pingcap#9622)
d098c73 
ref pingcap#9413
@yk133 yk133 mentioned this issue Aug 30, 2023
Merged
18 tasks
This was referenced Aug 30, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 5, 2023
@asddongmen
pulsar (ticdc): pulsar use ticdc log (#9674)
e1730e5 
ref #9413
ti-chi-bot bot pushed a commit that referenced this issue Sep 6, 2023
@asddongmen
pulsar (ticdc): add partition key for pulsar meesage (#9677)
254cc2b 
ref #9413
ti-chi-bot bot pushed a commit that referenced this issue Sep 8, 2023
@asddongmen
pulsar (ticdc): fix some pulsar sink related error (#9657)
b9a647d 
ref #9413
@yumchina yumchina mentioned this issue Sep 11, 2023
Merged
This was referenced Sep 11, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 15, 2023
@asddongmen
cmd (ticdc): add pulsar consumer (#9730)
d0a9a4b 
ref #9413
@asddongmen asddongmen mentioned this issue Sep 19, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 20, 2023
@asddongmen
pulsar (ticdc): fix pulsar sink related errors (#9770)
907d937 
ref #9413
ti-chi-bot bot pushed a commit that referenced this issue Sep 25, 2023
@yumchina
metrics, pulsar (ticdc): add pulsar metrics panel (#9728)
695e340 
ref #9413
@asddongmen
Copy link
Contributor

All development are finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@asddongmen asddongmen

@yumchina yumchina

Labels
area/ticdc Issues or PRs related to TiCDC. type/feature Issues about a new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@asddongmen @yumchina @nongfushanquan