Subscription to topics with regex not working for non-persisten topics

[itaross]

Messages on non-persistent topics are not received when using a regex subscription.
This problem is encountered when using a Pulsar IO custom connector (sink).

When using persistent topics the subscription is working as expected

Expected behavior
Records are received when subscribing to regex topics.

Additional context
Issues #2009
#2009

Issues #2025
#2025

Does not solve this problem

@merlimat @sijie

[codelipenghui]

@itaross Can you provide more details about the bug? what version are you using and how to reproduce it. This will help to troubleshoot this issue.

[itaross]

I am currently using version 2.5.1
I think you can reproduce it even with a builting connector.
Create it from the coommand line, using --topics-pattern and use a non-persistent:// based pattern.
You will see that no data is received in the write method.

This has been highlighted also in the previous cited issued, but it has not been addressed in sinks. Everything works if you you use persistent topics.

[nicolo-paganin]

@codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful

[wolfstudy]

@codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful

Can you provide a way to reproduce? In functions, sinks and sources, --topics-pattern will be deprecated, can you use --inputs replace it? [--input] and [--topic-pattern] are mutually exclusive.

[itaross]

@codelipenghui have you been able to reproduce? This is something that I think has never worked, but very useful

Can you provide a way to reproduce? In functions, sinks and sources, --topics-pattern will be deprecated, can you use --inputs replace it? [--input] and [--topic-pattern] are mutually exclusive.

@wolfstudy what you are referring to is probably this line https://github.com/oncodeit/pulsar/blob/71b3819bfc11afe0f0da19df0a19e3b7a05988c5/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java#L258
but this involves only the name of the pattern parameter

To reproduce: take any connector from Pulsar IO, a sink in this case.
https://archive.apache.org/dist/pulsar/pulsar-2.5.1/connectors/

Follow these steps: https://pulsar.apache.org/docs/en/standalone/#install-builtin-connectors-optional

Launch the connector with:

./pulsar-admin sinks create --tenant public --namespace default --name test-sink --sink-type jdbc  --sink-config-file <the yaml config file>  --topicsPattern "persistent://public/default/.*"

content of the yaml:

{
    "userName": "root",
    "password": "jdbc",
    "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink",
    "tableName": "pulsar_mysql_jdbc_sink"
}

then retry using:

./pulsar-admin sinks create --tenant public --namespace default --name test-sink --sink-type jdbc  --sink-config-file <the yaml config file>  --topicsPattern "non-persistent://public/default/.*"

You will notice that the connector will now works.

I posted this example with jdbc/mysql but you can try with anything you prefer, the behaviour is the same.

[itaross]

Hi @wolfstudy, any update on this? May I help with this?

[marksilcox]

@codelipenghui @wolfstudy Has there been any update on this? I am seeing the same in 2.9.1