Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc/sink: adjust kafka initialization logic #3192

Merged
merged 80 commits into from
Nov 12, 2021
Merged

cdc/sink: adjust kafka initialization logic #3192

merged 80 commits into from
Nov 12, 2021

Commits on Nov 12, 2021

  1. add admin file

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    f71d6aa View commit details
    Browse the repository at this point in the history

  2. rebase master.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    01f2a59 View commit details
    Browse the repository at this point in the history

  3. fix create topic.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    78b34dd View commit details
    Browse the repository at this point in the history

  4. use config.partitionNum directly.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    7487412 View commit details
    Browse the repository at this point in the history

  5. close admin.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    8accab1 View commit details
    Browse the repository at this point in the history

  6. fix testcase.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    e17169d View commit details
    Browse the repository at this point in the history

  7. tiny fix.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    2592d9a View commit details
    Browse the repository at this point in the history

  8. fix some typo.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    eec9a3d View commit details
    Browse the repository at this point in the history

  9. refactor parameter check in topicPreProcess.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    def3227 View commit details
    Browse the repository at this point in the history

  10. fix kafka test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    bc76e4a View commit details
    Browse the repository at this point in the history

  11. add a test for config initialization.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    4a8e731 View commit details
    Browse the repository at this point in the history

  12. remove unnecessary code in kafka test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    62b8eff View commit details
    Browse the repository at this point in the history

  13. add ut for initialization.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    4a780b3 View commit details
    Browse the repository at this point in the history

  14. use PartitionNum.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    57747c1 View commit details
    Browse the repository at this point in the history

  15. remove unnecessary kafka package at the moment.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    c9263ea View commit details
    Browse the repository at this point in the history

  16. add a log.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    75e9091 View commit details
    Browse the repository at this point in the history

  17. tiny fix.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    2f48945 View commit details
    Browse the repository at this point in the history

  18. tiny fix.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    068371a View commit details
    Browse the repository at this point in the history

  19. use failpoint to workaround for NewClusterAdmin in test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    4924f90 View commit details
    Browse the repository at this point in the history

  20. simple typo fix.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    69d3c1b View commit details
    Browse the repository at this point in the history

  21. add more test for preprocess.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    a0f919a View commit details
    Browse the repository at this point in the history

  22. disable the checkpoint.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    43a523e View commit details
    Browse the repository at this point in the history

  23. fix enable failpoint syntax.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    4f1f032 View commit details
    Browse the repository at this point in the history

  24. fix parameters.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    da10398 View commit details
    Browse the repository at this point in the history

  25. Adjust the layout of comment for create sarama config.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    2f2e780 View commit details
    Browse the repository at this point in the history

  26. adjust layout for sarama config producer.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    4d73caa View commit details
    Browse the repository at this point in the history

  27. fix test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    f371a32 View commit details
    Browse the repository at this point in the history

  28. fix kafka version in mq_test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    7396f02 View commit details
    Browse the repository at this point in the history

  29. remove failpoint.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    268519a View commit details
    Browse the repository at this point in the history

  30. remove control by variable.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    41a93ea View commit details
    Browse the repository at this point in the history

  31. add failpoint.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    004e817 View commit details
    Browse the repository at this point in the history

  32. add failpoint.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    add6c07 View commit details
    Browse the repository at this point in the history

  33. fix failpoint interm.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    58d40e9 View commit details
    Browse the repository at this point in the history

  34. revert version.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    c873264 View commit details
    Browse the repository at this point in the history

  35. try to fix max-message-size larger than topic.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    c65b5f0 View commit details
    Browse the repository at this point in the history

  36. add a debug log.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    8f8d9aa View commit details
    Browse the repository at this point in the history

  37. remove the log.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    8dee36c View commit details
    Browse the repository at this point in the history

  38. create topic with max-message-size.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    dec15c0 View commit details
    Browse the repository at this point in the history

  39. fix comment.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    8217ae3 View commit details
    Browse the repository at this point in the history

  40. Update cdc/sink/producer/kafka/kafka.go 

    Co-authored-by: Neil Shen <overvenus@gmail.com>
    @3AceShowHand @overvenus
    3AceShowHand and overvenus committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    98977fd View commit details
    Browse the repository at this point in the history

  41. fix by code review suggestion.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    be59f9b View commit details
    Browse the repository at this point in the history

  42. Update cdc/sink/producer/kafka/kafka.go 

    Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
    @3AceShowHand @liuzix
    3AceShowHand and liuzix committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    b49cf35 View commit details
    Browse the repository at this point in the history

  43. Update cdc/sink/producer/kafka/kafka.go 

    Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
    @3AceShowHand @liuzix
    3AceShowHand and liuzix committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    7276058 View commit details
    Browse the repository at this point in the history

  44. Update cdc/sink/producer/kafka/kafka.go 

    Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
    @3AceShowHand @liuzix
    3AceShowHand and liuzix committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    388e42c View commit details
    Browse the repository at this point in the history

  45. fix by code review suggestion.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    e495cb8 View commit details
    Browse the repository at this point in the history

  46. fix log layout.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    46d1243 View commit details
    Browse the repository at this point in the history

  47. make topic out of the Config.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    72d05d5 View commit details
    Browse the repository at this point in the history

  48. fix test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    91ea692 View commit details
    Browse the repository at this point in the history

  49. fix test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    73f7175 View commit details
    Browse the repository at this point in the history

  50. fix by code review suggestion.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    0eb4547 View commit details
    Browse the repository at this point in the history

  51. Rewrite the logic in topicPreProcess.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    ba707c4 View commit details
    Browse the repository at this point in the history

  52. Adjust the layout of comment for create sarama config.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    02f5300 View commit details
    Browse the repository at this point in the history

  53. fix logic if topic already exist.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    e1e9d27 View commit details
    Browse the repository at this point in the history

  54. Adjust the layout of comment for create sarama config.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    a0b8611 View commit details
    Browse the repository at this point in the history

  55. tiny fix.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    ab525f7 View commit details
    Browse the repository at this point in the history

  56. remove useless test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    0957074 View commit details
    Browse the repository at this point in the history

  57. try to get broker configuration.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    8bb0775 View commit details
    Browse the repository at this point in the history

  58. add print config entries.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    26cf591 View commit details
    Browse the repository at this point in the history

  59. fix convert to string.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    3466091 View commit details
    Browse the repository at this point in the history

  60. tiny fix.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    a158a64 View commit details
    Browse the repository at this point in the history

  61. move get broker config up.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    0386d93 View commit details
    Browse the repository at this point in the history

  62. refine the logic when create the topic.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    614cc5e View commit details
    Browse the repository at this point in the history

  63. remove initialize topic with max-message-bytes.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    e536a7c View commit details
    Browse the repository at this point in the history

  64. adjust comment.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    7e8aa4b View commit details
    Browse the repository at this point in the history

  65. fix the comment.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    3ceb5bf View commit details
    Browse the repository at this point in the history

  66. fix the comment.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    6b6c4c4 View commit details
    Browse the repository at this point in the history

  67. only fetch message.max.bytes.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    cf7e0ca View commit details
    Browse the repository at this point in the history

  68. refine the comment.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    b8852ae View commit details
    Browse the repository at this point in the history

  69. if topic cannot find the max.message.bytes, find message.max.bytes fr… 

    …om brokers.
    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    dcb8562 View commit details
    Browse the repository at this point in the history

  70. extract adjustPartitionNum.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    5d95d75 View commit details
    Browse the repository at this point in the history

  71. remove TopicPreProcessCreate test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    f24bc4e View commit details
    Browse the repository at this point in the history

  72. revert comment out test.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    16e5c90 View commit details
    Browse the repository at this point in the history

  73. check max-message-bytes only for default and craft protocol.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    cadfc72 View commit details
    Browse the repository at this point in the history

  74. fix make check.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    5603c5a View commit details
    Browse the repository at this point in the history

  75. fix error handling.

    @3AceShowHand
    3AceShowHand committed Nov 12, 2021
    Configuration menu
    Copy the full SHA
    9e94a61 View commit details
    Browse the repository at this point in the history

  76. Update cdc/sink/mq.go 

    Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
    @3AceShowHand @liuzix
    3AceShowHand and liuzix authored Nov 12, 2021
    Configuration menu
    Copy the full SHA
    9d5b6a2 View commit details
    Browse the repository at this point in the history

  77. Merge branch 'master' into extract-kafka-admin

    @ti-chi-bot
    ti-chi-bot authored Nov 12, 2021
    Configuration menu
    Copy the full SHA
    bad0771 View commit details
    Browse the repository at this point in the history

  78. Merge branch 'master' into extract-kafka-admin

    @ti-chi-bot
    ti-chi-bot authored Nov 12, 2021
    Configuration menu
    Copy the full SHA
    8cc9b91 View commit details
    Browse the repository at this point in the history

  79. Merge branch 'master' into extract-kafka-admin

    @ti-chi-bot
    ti-chi-bot authored Nov 12, 2021
    Configuration menu
    Copy the full SHA
    26b770c View commit details
    Browse the repository at this point in the history

  80. Merge branch 'master' into extract-kafka-admin

    @ti-chi-bot
    ti-chi-bot authored Nov 12, 2021
    Configuration menu
    Copy the full SHA
    1e2b3c7 View commit details
    Browse the repository at this point in the history