Improve remove_markup handling of Wikipedia headers

[marat-s]

Fix #2561

[gojomo]

The link to the issue in the PR description is good - but ideally the title will also describe the matters affected.

But also: see my comment on #2561 – despite the original reporter's preference that headings be stripped, I doubt that's the right default for everyone.

[mpenkov]

I've taken the liberty of adjusting the title.

@gojomo Perhaps we can include a strip_header_markup keyword parameter (defaults to False) to handle both sides of the argument?

[gojomo]

@mpenkov A parameter-controlled option would certainly support the needs of the reporter, @0x0badc0de, and address my concern about inconveniencing others.

But this need might be narrow enough that the right advice, to those who need it, is to run the ~1-liner to strip headers yourself if you need it. (Roughly just a s/^(=+).*\1// regex replacement on the output text.)

[gojomo]

See the further discussion on #2561 for clarification on what kind of fix the original reporter was expecting, and would be best.

[gojomo]

Does this retain the words of the heading? (I don't think so.)

[SagarDollin]

Hi @gojomo,
The re.sub function using RE_P18 does retain the words in the heading. Please refer to the screenshot attached where I test if this works on a jupyter notebook.

A comment below asks for unit tests; If you could guide me, I could do that and fix any issues that could have been introduced due to the changes. I would appreciate any pointers and your expectations from the tests.

[mpenkov]

Please add unit tests for your new functionaity.

[SagarDollin]

Please add unit tests for your new functionaity.

Hey @mpenkov, I could test this. Would you please guide me to your expectations from the unit test. And also any contribution guidelines before creating another PR. You can view a small test that I conducted here.