Add a new ParserExtra::State: Recorder trait
#681
Conversation
This allows using imperative concrete syntax tree parsers like `rowan` and `cstree`.
In particular those libraries want to know every token that is parsed, and need to know when chumsky backtracks and reparses the same tokens again.
This adds the following new API surface:
```rust
pub trait Recorder<'a, I: Input<'a>>: Default {
type SaveMarker: Copy + Clone;
fn on_token(&mut self, token: I::Token);
fn on_save<'parse>(&self, offset: I::Offset) -> Self::SaveMarker;
fn on_rewind<'parse>(&mut self, marker: Marker<'a, 'parse, I, Self::SaveMarker>);
}
pub struct SimpleState<T>(pub T);
impl<'a, T, I: Input<'a>> Recorder<'a, I> for SimpleState<T>;
impl<T> DerefMut<Target = T> for SimpleState<T>;
impl<T> From<T> for SimpleState<T>;
```
and additionally now requires `ParserExtra::State: Recorder`.
jyn514
changed the title
ParserExtra::State::Recorder]ParserExtra::State: Recorder trait
|
|
||
| /// A type that receives event hooks when certain parsing actions occur. | ||
| /// | ||
| /// Unlike [`Parser::State`], this receives information about the incoming actions. |
There was a problem hiding this comment.
This line seems irrelevant now.
| #[allow(unused)] // for intra-doc links | ||
| use crate::Parser; | ||
|
|
||
| /// A type that receives event hooks when certain parsing actions occur. |
There was a problem hiding this comment.
I think I'd be tempted to just call this type State and remove the existing type alias(es) (State, Context, and Err). Recorder seems too specific a name for what this might eventually turn into.
| type SaveMarker: Copy + Clone; | ||
|
|
||
| /// This function is called when a new token is read from the input stream. | ||
| fn on_token(&mut self, token: I::Token); |
There was a problem hiding this comment.
Is this ever getting invoked? I assume not, since this sadly doesn't generalise to by-ref and by-value inputs. The correct thing would be token: I::TokenMaybe.
There was a problem hiding this comment.
FWIW, the place to call it is in these functions:
InputRef::skip_whileInputRef::next_innerInputRef::next_maybe_innerInputRef::next_ref_inner
| impl<T> From<T> for SimpleState<T> { | ||
| fn from(value: T) -> Self { | ||
| Self(value) | ||
| } | ||
| } |
There was a problem hiding this comment.
Should probably also add From<SimpleState<T>> for T (does this break the orphan rules nowadays, or will it need to be Into? I can't remember).
This allows using imperative concrete syntax tree parsers like
rowanandcstree. In particular those libraries want to know every token that is parsed, and need to know when chumsky backtracks and reparses the same tokens again.This adds the following new API surface:
and additionally now requires
ParserExtra::State: Recorder.i didn't write any of my own tests yet but my own parser that hooks chumsky up to cstree does work: spreadsheet-lang/spreadsheet@1b97313#diff-14962dfdd56397d99e07041d6337cb5b0327f4d7433d80fd3383955415014910R111
Fixes #96