Fix brackets in links in markdown lexer

[Ravlen]

When hiding brackets inside links with backticks, rendering
was breaking. This makes the lexer search for ][ or ](
to find the end delimiting the link text.

This fixes #1444.

[Ravlen]

@pyrmont I see your lookahead suggestion, but it wasn't working when I tested. I'll give it another go and see if I can figure it out, and I'll update this PR if I can get it to work. It might just be the regex tester I'm using doesn't like the lookahead formatting. I understand the logic now.

Also, does this need to add an example to the spec, like you did in #1442?

[Ravlen]

@pyrmont OK, sorry about that, I think it's good to go now. 🙏

[Ravlen]

@pyrmont Thanks a lot for the fixes! Feel free to point them out to me next time, and I'll fix them myself so you don't have to go through all the effort. It also means I can fix my commit, so I'm not pushing broken regex to the project, even if it is fixed right after. 🙇

I've also cloned locally and am able to test, (super simple, works great), so I'll test locally myself before submitting next time. 🙇

[Ravlen]

@pyrmont I don't think the regex you pushed was correct, as there shouldn't be any spaces between the link text and the URL/Reference:

[This is a link `with backticks`](example.com)
[This is not a link `with backticks`] (example.com)

By adding spaces/tabs to be allowed between the ] and ( or [, it broke:

I removed that in the latest commit, and now it seems correct:

I also added that example to the markdown sample spec. What do you think?

[pyrmont]

A couple of things:

1. Don't worry about fixing broken commits. It's preferable to leave them in, to be honest. We follow a squash and merge strategy when merging so it won't pollute the eventual commit history and it means the true development process is easier to follow if/when someone is looking at the PR.

2. Markdown can have spaces between the text and the reference (see here). Strictly speaking, space is only permitted for reference links but that complicates the logic and I'm not convinced the lazy approach would lead to false positives in real life. Syntax correctness is a non-goal of Rouge so being more permissive than is correct syntactically isn't itself a problem.

What do you think?

[Ravlen]

@pyrmont In this case, I'm trying to match the behavior I see most commonly.

Testing this markdown:

### Try links

- [This is a link](https://www.google.com)
- [This is not a link] (https://www.google.com)
- [This is a link][example]
- [This is a link?] [example]

[example]: https://www.google.com

Testing with CommonMark dingus

Testing with GitHub:

Try links

• This is a link
• [This is not a link] (https://www.google.com)
• This is a link
• [This is a link?] example

Testing with GitLab

Testing with VS Code

It seems like the majority of renderers do not accept this, including commonmark. It's not surprising because I can see how it would cause ambiguity or confusion when mixing references. WDYT?

[pyrmont]

Agreed that not putting a space between the link and the reference is the approach everyone else is taking. Let's do that.

[pyrmont]

@Ravlen Thanks for everything with this PR! I hope this improves the link highlighting behaviour on your end :)