[improve][pip] PIP-346: Add a simplified configuration file for Pulsar

[thetumbled]

Motivation

PIP-346

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[crossoverJie]

How do we confirm which configurations need to be modified frequently?

[BewareMyPower]

I think it depends on the experience.

[thetumbled]

We can vote the configurations in the implementation pr.
It is a process of compression of operation and maintenance experience, which is valuable in my view.

[merlimat]

I think this is a good discussion starting point, though the precise list and modality should be eventually part of the PIP that is voted on.

[BewareMyPower]

LGTM

[dao-jun]

I support

[thetumbled]

Hi, i have added the candidate conf list, looking forward to your feedback. @merlimat @BewareMyPower @crossoverJie @dao-jun

[lhotari]

I provided my feedback on the mailing list https://lists.apache.org/thread/xdq62jytxxj421vly6qzn9dn8vvjqyv1

Before making this change, I hope that there could be discussion about
improving the configuration solution in Pulsar.
The mailing list thread is
https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh .

With a proper configuration library such as Gestalt Config [1], it's
possible to have a solution that is expressed in this PIP-346.
In Gestalt Config, the feature "Merge Multiple Sources: Seamlessly
merge configurations from different sources to create comprehensive
settings." support this.

This is the reason why I don't think that we should move forward with
PIP-346 without addressing the proposal in the other mailing list
thread
"[DISCUSSION] Proposal to Replace Pulsar's Homegrown Configuration
Framework with Gestalt Config" [2].

-Lari

1 - https://github.com/gestalt-config/gestalt#features
2 - https://lists.apache.org/thread/8splwyrn25pt5rb6ph0yz0tfgknn3hqh

[thetumbled]

Hi, could we move forward with this PIP? It seems that it is not rely on another PIP Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config.
@merlimat @BewareMyPower @crossoverJie @codelipenghui @dao-jun @lhotari

[lhotari]

Hi, could we move forward with this PIP? It seems that it is not rely on another PIP Proposal to Replace Pulsar's Homegrown Configuration Framework with Gestalt Config. @merlimat @BewareMyPower @crossoverJie @codelipenghui @dao-jun @lhotari

The current Pulsar configuration solution is currently a mess (example: apply-config-from-env.py is needed to override configuration from environment) and instead of just doing changes that don't really address the problems, I think that it's a distraction.

That's why I think that we shouldn't proceed on PIP-346. We need a design for revisiting the configuration solution for Pulsar 4.0 and beyond.