Support for iterator-range parsing

[theodelrieu]

Hi, it would be handy to add an overload to the parse method.

In my code, I deal a lot with vector<uint8_t> when communicating with my server.
However I sometimes have to send JSON, and I must convert the buffer to a std::string, which is unfortunate.

I know I could typedef basic_json and use std::basic_string<uint8_t> as StringType.
However, other parts of the code uses the std::string version, and I don't want to patch the whole code because of that.

We could add this overload to parse, which will behave like std::string iterator constructor:

template<typename InputIterator>
static basic_json parse(InputIterator begin, InputIterator end, parser_callback_t cb = nullptr);

// user code

auto buffer = receive_bytes();
auto json = json::parse(buffer.begin(), buffer.end());

What are your thoughts on that ?
If you agree with the idea, I can take care of the PR.

[nlohmann]

I think the current implementation would not benefit from an iterator-range access: Inside the parser, I employ a lexer which relies on access to the input via a pointer of type lexer_char_t* (unsigned char*). For std::string, this can be done with casting the result of c_str(), and std::istreams are copied line by line to a buffer of type std::string.

To process a std::vector<uint8_t>, such a pointer could be casted from data(). Maybe one could generalize the static basic_json parse(const string_t& s, const parser_callback_t cb = nullptr) function to

template<typename T>
static basic_json parse(const T& s, const parser_callback_t cb = nullptr)

where T has a data() member function (std::basic_string, std::vector, and std::array) and whose pointers to its value type can be casted to lexer_char_t*.

What do you think?

[theodelrieu]

So only contiguous containers of 1 byte integrals are supported I believe? (for now at least)
I don't know if counting on the data() method is a good thing, what if an user gives a homemade vector, or a char[] ?

Maybe it's better to use methods defined in the Container Concept, and use the std::begin method. We could static_assert that the iterator type must be RandomAccessIterator to ensure the contiguity of the container.

It could look like this:

template<typename ByteContainer>
static basic_json parse(const ByteContainer& buf, const parser_callback_t cb = nullptr)
{
  // ensure std::begin(ByteContainer) is valid + retrieve iterator type
  using Traits = std::iterator_traits<decltype(std::begin(std::declval<ByteContainer>()))>;

  static_assert(sizeof(Traits::value_type) == 1, "");
  // There is no new tag for ContiguousIterator in C++17, won't need to change
  static_assert(std::is_same_v<Traits::iterator_category, std::random_access_category_tag>, "");

  // quite ugly..., but will work with containers, C arrays
  do_lexing(reinterpret_cast<const lexer_char_t*>(&(*std::begin(buf)));
}

I think it's reasonable to assume that the container is invalid if iterator_traits cannot be defined with it.
That's also an issue on LWG.

What's your opinion ?

[theodelrieu]

By the way, we can check for contiguity with the Iterator overload too, which removes some declval horror, so I think the iterator prototype is still correct.

Maybe we could add both ?

[nlohmann]

This looks good!

So far, parsing a string or an input stream could rely on an end of string or end of file mark. For this overload, one would need to pass a length to the parser - I guess std::distance(std::begin(buf), std::end(buf)) would do the job.

Shall I open a feature branch for this?

[theodelrieu]

Sure!

You want the user to give the length to parse?
Could you show me what prototype you have in mind?

[nlohmann]

If std::distance can find out the length, then there is no need for the user to provide the length. (Unless we want to allow for a totally generic parse(const char*).

I do not have a prototype in mind. I would just implement and integrate your proposal into a feature branch so we can find out whether we oversaw a dependency.

[theodelrieu]

Oh ok, I misunderstood what you wanted.
If a user wants to use parse with a const char*, I think the contiguous iterator-range overload does the job pretty well

[nlohmann]

I started a feature branch and started with a constructor for the parser class that takes an iterator range, see https://github.com/nlohmann/json/blob/feature/iterator_range_parsing/src/json.hpp.re2c#L8195.

It currently only checks if the iterators are random-access. This is, however, not sufficient to guarantee contiguous storage, and I have not found a way to enforce this in C++11. However, first tests (see https://github.com/nlohmann/json/blob/feature/iterator_range_parsing/test/src/unit-class_parser.cpp#L748) show that parsing works with

• std::vector
• std::array
• std::string
• std::initializer_list
• std::valarray
• C-style arrays

The next step would be to add overloads for the parse function for the basic_json class.

Any comments?

[gregmarr]

This is, however, not sufficient to guarantee contiguous storage, and I have not found a way to enforce this in C++11.

This is not a static property, but a runtime property, unfortunately. For example, std::deque can be either contiguous or not depending on the size of the deque.

http://stackoverflow.com/questions/35004633/checking-if-a-sequence-container-is-contiguous-in-memory

[nlohmann]

@gregmarr Thanks for clarifying. So I think the only way to cope with this is to make clear in the documentation that the behavior is undefined for iterator ranges with non-contiguous storage. Or any better idea?

[gregmarr]

Probably. You could probably have an assert that you get the same address for begin iterator plus range length, and the end iterator.

[nlohmann]

I added an assertion to do the check from the Stack Overflow article:

int i = 0;
assert(std::accumulate(first, last, true, [&i, &first](bool res, decltype(*first) val)
{
    return res and (val == *(std::next(std::addressof(*first), i++)));
}));

(Any idea how to get the int i = 0; inside the lamdba? Making it a static variable did not work)