JsonEvent::ArrayIndex

[scott-wyatt]

Hi @Tpt, before I fork and just do this in my own version, would you be open to an event for an Array index? It wouldn't be a stateful index (it shouldn't have that in the library) but there should be a signal that a new array index is being opened, just like there is for an ObjectKey event.

This is useful for many things:

• Splitting large arrays
• Counting indexes
• etc.

[Tpt]

Hi, thank you for your interest in Oxigraph. I am not sure to fully grasp what is the use case for it. May you give me an example 🥺

One need to maintain a stack anyway to know when an array/object is being open closed so matching on the ObjectKey event or on String(_) | Number(_) | Boolean(_) | Null | StartObject | EndObject seems to provide the exact same information to me. But I am maybe missing something

[scott-wyatt]

Thanks for getting back @Tpt .

The events for String/Number/Boolean/Null are ambiguous to:
a) a json document that contains only one element (which is valid JSON)
b) An Object Value and Array Value

So the only way to know when you are seeing an array value (and no other type of value) event is to create a state in the application (by tracking whether the current depth was last an object or an array, this is a LOT of state to manage). Even in the json-event-stream code, array values are un-ambiguous from object-values during parsing. (So if that doesn't prove its usefulness, I don't know what will haha.)

The library currently emits an ObjectKey event (which it has too) because a key is stateful, an ArrayIndex event could be a cheap stateless event. Many applications will ignore it, but there are a variety of applications that can benefit, here are some examples:

1. Partitioning large arrays. Some GeoJSON apps have GIGANTIC arrays, where pairs of values can be sorted into different threads.
2. Validating array length. Sometimes you may not care about the values, only that there only so many.
3. Streaming vectors. Quick counters to push floating point numbers into a buffer while only managing a usize instead of a scope tree is a huge advantage (this is one of my use cases).
4. If I just am only tracking ObjectKey or ArrayIndex (instead of managing a whole stack), then I can know the parent with very little overhead. This also helps me with feature to drain the next value from the buffer #35

[Tpt]

Thank you for this detailed reply.

So the only way to know when you are seeing an array value (and no other type of value) event is to create a state in the application (by tracking whether the current depth was last an object or an array, this is a LOT of state to manage).

I am not sure to see how you can escape this with ArrayIndex. Indeed, without a stack there is no way to know if the ArrayIndex belongs to the array you are interested in or an other array nested into it. So, you need to maintain a stack anyway.

An other API that might be way more convenient might be to have a small wrapper on top if the current low level parser that would expose the current stack. For example in the followin JSON:

{
   "foo": [{"bar": [1,2]}]
}

when reading the number 2 the stack would be [Object("foo"), Array(0), Object("bar"), Array(1)]

What do you think about it?

[scott-wyatt]

I am not sure to see how you can escape this with ArrayIndex

This is in general a fair point. I'd argue that it's semantically correct to emit an index event, as it is what is happening (I'm parsing JSON into a tuple language).

I'm trying to avoid collecting state where ever possible. However, I'm using this mostly in conjunction with of #35, where I want to take the next value and push it to a different parser.

What it seems to me that you are suggesting is like a pseudo "json path" implementation, I don't think it should be included in the base API. Although a "ObjectValue or ArrayValue" would also work for me 😄.

Personally, I have an implementation of this that uses a specific path, and this would be code bloat for me. So maybe bindings, but not directly in the API? Or a trait that could be implemented?

Why I chose this library was because it was stateless by default, it'd be a shame to deviate from that!

[Tpt]

I'd argue that it's semantically correct to emit an index event

Yes, I agree with that. Point is more that it would not be really useful outside of simplistic senarios (the flat array) and so it's not worth it to complexify the API for it.

I don't think it should be included in the base API.

Yes! It's why I think it should be a wrapper on top of the existing API named something like "PositionAwareParser".

[scott-wyatt]

Yes, I agree with that. Point is more that it would not be really useful outside of simplistic senarios (the flat array) and so it's not worth it to complexify the API for it.

On the contrary, in the scenario of "drain next value", you'll need an Array Index event. Luckily, it took like 3 lines to implement it, because the logic was essentially already there, just no event.

Regardless, it's going to be the point at which I'll need to hard fork this repo or not. I'll push upstream other fixes and those drain features if they make sense.

[Tpt]

On the contrary, in the scenario of "drain next value", you'll need an Array Index event.

I guess you need it to have know that there is a next array element before starting to read it? In this case it means you don't care about the element value and it is likely you want to drain the full array or am I missing something?
Sorry for being annoying, I am still not understanding why there is a need for it. I already wrote "drain next value" in parsers on top of json-event-parser and have not seen the need for it.

Regardless, it's going to be the point at which I'll need to hard fork this repo or not. I'll push upstream other fixes and those drain features if they make sense.

Feel free to fork if you want to move at your own speed, I am quite overwhelmed right now.

[scott-wyatt]

I guess you need it to have know that there is a next array element before starting to read it

This precisely. I do not want to drain the whole array, just the next value 😂. You may not deal with arrays with 1m+ entries, but that's daily for me.

I already wrote "drain next value" in parsers on top of json-event-parser and have not seen the need for it.

Is this available somewhere?

Feel free to fork if you want to move at your own speed, I am quite overwhelmed right now

I'm honestly sorry to hear that, it happens to us all. Hope the best for you!

[Tpt]

This precisely. I do not want to drain the whole array, just the next value 😂
Can you do it with something like (not real Rust code):

/// Returns true if there is no next value and it's the end of the array
fn skip_next_value<R>(parser: ReaderJsonParser<R>) > bool {
    let mut depth = 0;
    loop {
        match parser.parse_next() {
            JsonEvent::String(_) | JsonEvent::Number(_) | JsonEvent::Boolean(_) | JsonEvent::Null => { if depth == 0 { return false; } },
            JsonEvent::StartArray | JsonEvent::StartObject => { depth += 1 },
            JsonEvent::EndArray => {
                  if depth == 0 { return true; } // depth already at 0, end of the root array
                  depth -= 1;
                  if depth == 0 { return false; } // depth was 1, closing nested array
            }, 
            JsonEvent::EndObject => { depth -= 1; if depth == 0 { return false; } },
            JsonEvent::ObjectKey(_) => (),
            JsonEvent::EOF => unreachable!()
        }
   }
}

Is this available somewhere?

Nothing clean, it's in a working branch. Here is some code to convert even into a JSON tree (it loops over the event with a state stack, for each event it pops the top of the stack and might push some states on top of it. after_to_node_event is called when the full tree is parsed).
Here is also some code that just skips.