Skip to content

Commit

Permalink
[Bug][KafkaSource]Failed to parse offset format (apache#3810)
Browse files Browse the repository at this point in the history
* When the kafka source is started in offset mode, when the topicName has a "-", the exception will be resolved, causing the assignment to fail

* fix code style.

* update change-log

Co-authored-by: zhaoliang01 <zhaoliang01@58.com>
  • Loading branch information
lightzhao and zhaoliang01 authored Dec 28, 2022
1 parent 39dae4c commit 8e1196a
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 2 deletions.
1 change: 1 addition & 0 deletions docs/en/connector-v2/source/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,3 +210,4 @@ source {

- [Improve] Support setting read starting offset or time at startup config ([3157](https://github.com/apache/incubator-seatunnel/pull/3157))
- [Improve] Support for dynamic discover topic & partition in streaming mode ([3125](https://github.com/apache/incubator-seatunnel/pull/3125))
- [Bug] Fixed the problem that parsing the offset format failed when the startup mode was offset([3810](https://github.com/apache/incubator-seatunnel/pull/3810))
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,11 @@ public void prepare(Config config) throws PrepareFailException {
Map<TopicPartition, Long> specificStartOffsets = new HashMap<>();
ObjectNode jsonNodes = JsonUtils.parseObject(offsetsJson);
jsonNodes.fieldNames().forEachRemaining(key -> {
String[] topicAndPartition = key.split("-");
int splitIndex = key.lastIndexOf("-");
String topic = key.substring(0, splitIndex);
String partition = key.substring(splitIndex + 1);
long offset = jsonNodes.get(key).asLong();
TopicPartition topicPartition = new TopicPartition(topicAndPartition[0], Integer.valueOf(topicAndPartition[1]));
TopicPartition topicPartition = new TopicPartition(topic, Integer.valueOf(partition));
specificStartOffsets.put(topicPartition, offset);
});
this.metadata.setSpecificStartOffsets(specificStartOffsets);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.seatunnel.connectors.seatunnel.kafka;

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

public class KafkaStartOffsetTest {

@Test
void getTopicNameAndPartition(){
String topicName = "my-topic-test";
int partIndex = 1;
String key = "my-topic-test-1";
int splitIndex = key.lastIndexOf("-");
String topic = key.substring(0, splitIndex);
String partition = key.substring(splitIndex + 1);
Assertions.assertEquals(topic, topicName);
Assertions.assertEquals(Integer.valueOf(partition), partIndex);
}
}

0 comments on commit 8e1196a

Please sign in to comment.