Parsing block comments in v1.0.0.alpha.7

[Divy1211]

Hi, I'm trying to parse comments using:

use chumsky::error::Rich;
use chumsky::prelude::*;

// Token is an enum
use crate::parsing::lexer::token::Token;
// type Span = SimpleSpan<usize>;
use crate::parsing::span::Span;

pub fn comment<'src>() -> impl Parser<
    'src, &'src str, Token, extra::Err<Rich<'src, char, Span>>
> {
    let line_comment = just("//").ignore_then(none_of("\r\n").repeated());
    let block_comment = just("/*").ignore_then(none_of("*/").repeated());
    
    line_comment.or(block_comment)
        .to_slice()
        .map(|val: &str| Token::Comment(String::from(val)))
}

But I recently found out that the code for block comment none_of("*/").repeated() is incorrect (I had misunderstood the behaviour of none_of) and that it breaks when comments contain * or /. I looked at 0.9.3 documentation by accident and came across a primitive called take_until which I tried using:

    let line_comment = just("//").ignore_then(take_until(text::newline()));
    let block_comment = just("/*").ignore_then(take_until(just("*/")));

but take_until is not a primitive in 1.0.0.alpha.7. Is there an alternative way to parse comments?

I tried implementing a custom parser, but couldn't figure out the proper syntax:

pub fn comment<'src>() -> impl Parser<
    'src, &'src str, Token, extra::Err<Rich<'src, char, Span>>
> {
    let line_comment = just("//").ignore_then(none_of("\r\n").repeated());
    let block_comment = just("/*").ignore_then(custom::<_, &str, _, _>(|inp| {
        let mut fslash_is_end = false;
        loop {
            match inp.next() {
                Some('/')  if fslash_is_end => { return Ok(()); }
                Some('*') => { fslash_is_end = true; }
                Some(___) => { fslash_is_end = false; }
                None => { Err(Rich::custom(/* how to determine span? */, "Unexpected End of Input")) }
            }
        }
    }))

    line_comment.or(block_comment)
        .to_slice()
        .map(|val: &str| Token::Comment(String::from(val)))
}

[zesterer]

You're probably looking for and_is and not. You can use them together like:

any().and_is(just("*/").not())

This tells chumsky to accept a character that is not the start of the sequence */.

Note that I think you might misunderstand what none_of does: like one_of, it accepts a set of characters to reject, not a sequence. So none_of("*/") will accept any character except * and /.

[Divy1211]

I was trying:

        just("/*").ignore_then(
            just("*/").not().repeated()
        ).then_ignore(just("*/"));

and my lexer was panicking... found Repeated combinator making no progress at .... So close! I seem to have missed the docs of not() stating that it is intended to be used with and_is() and that not() won't consume input. No wonder the above code doesn't work.

Note that I think you might misunderstand what none_of does: like one_of, it accepts a set of characters to reject, not a sequence. So none_of("*/") will accept any character except * and /.

yes, that was precisely my misunderstanding initially haha, that's why I was looking for another way to do what I needed, but I couldn't figure out exactly which primitive/combinator combination to use.

This works like a charm!

        just("/*").ignore_then(
            any().and_is(just("*/").not()).repeated()
        ).then_ignore(just("*/"));

Thanks so much for the help! =)

I did also figure out how to use the custom parser but this is much cleaner