jsonpointer: flatten/unflatten does not preserve arrays with more than 10 elements #644
Description
Describe the bug
We encountered a problem with the jsonpointer extension when used with sorted objects json/wjson. When an array is flattened and then unflattened it is preserved only if its size is 10 or less.
Include a small, self-contained example if possible
#include "jsoncons/json.hpp"
#include "jsoncons_ext/jmespath/jmespath.hpp"
#include <jsoncons_ext/jsonpointer/jsonpointer.hpp>
#include <jsoncons_ext/jsonpointer/jsonpointer_error.hpp>
#include <iostream>
#include <string>
inline std::string PrettyPrint(const jsoncons::json &data) {
std::ostringstream oss;
oss << jsoncons::pretty_print(
data, jsoncons::json_options{}.indent_size(2).spaces_around_comma(
jsoncons::spaces_option::no_spaces));
return oss.str();
}
int main()
{
std::string src = R"({
"arr": ["0", "1", "2", "3", "4", "5", "6","7","8", "9", "10", "11"]
})";
try
{
auto result = jsoncons::json::parse(src);
std::cout << PrettyPrint(result) << std::endl;
std::cout << std::endl;
auto flatResult = jsoncons::jsonpointer::flatten_with_padded_array_indexes(result);
std::cout << PrettyPrint(flatResult) << std::endl;
std::cout << std::endl;
auto unflatten = jsoncons::jsonpointer::unflatten(flatResult);
std::cout << PrettyPrint(unflatten) << std::endl;
}
catch (const std::exception& e)
{
std::cout << "Failed: " << e.what() << std::endl;
return 1;
}
return 0;
}
What compiler, architecture, and operating system?
Visual Studio 2022 on Windows 11,
Toolset: Visual Studio 2022 (v143),
C++ Language Standard: Preview - ISO C++23 Standard (/std:c++23preview)
What jsoncons library version?
Latest release: 1.4.3
Additional notes
One issue was found in the try_unflatten_array implementation which was trying to match sequential values when it enumerates and parses the indexes - the next it finds after 1 is 10, not 2 .
...
auto s = *it;
std::size_t n{0};
auto r = jsoncons::utility::dec_to_integer(s.data(), s.size(), n);
if (r.ec == std::errc() && (index++ == n))
{
if (!part->is_array())
{
*part = Json(json_array_arg);
}
,,,
This implementation can be easily modified to check that, after enumerating all items, the maximum index value is equal to the number of parsed indexes minus 1. This will ensure that the array type is preserved but the original order of the elements in the array will not be preserved,
One possible way to preserve the array element order could be to provide an option for the flatten operation to pad the indexes with leading zeroes so that their original relative order is preserved. I played a bit with this idea and it worked nicely:
Added extra parameter bool with_padded_array_indexes to flatten_ and modified the implementation to pad indexes with leading zeroes when the parameter is set to true
...
size_t decimal_positions(size_t value)
{
size_t pos = 0u;
while (value > 0)
{
++pos;
value /= 10;
}
return pos;
}
template <typename Json>
void flatten_(const std::basic_string<typename Json::char_type>& parent_key,
const Json& parent_value,
Json& result,
bool with_padded_array_indexes)
{
using char_type = typename Json::char_type;
using string_type = std::basic_string<char_type>;
switch (parent_value.type())
{
case json_type::array_value:
{
if (parent_value.empty())
{
// Flatten empty array to null
//result.try_emplace(parent_key, null_type{});
//result[parent_key] = parent_value;
result.try_emplace(parent_key, parent_value);
}
else
{
auto max_size = parent_value.size();
string_type padding_zeroes(with_padded_array_indexes ? decimal_positions(max_size) : 0, 0x30);
for (std::size_t i = 0; i < max_size; ++i)
{
if ( 0 == i % 10 && !padding_zeroes.empty())
{
padding_zeroes.resize(padding_zeroes.size()-1);
}
string_type key(parent_key);
key.push_back('/');
if (!padding_zeroes.empty())
{
key += padding_zeroes;
}
jsoncons::utility::from_integer(i,key);
flatten_( key, parent_value.at(i), result, with_padded_array_indexes);
}
}
break;
}
...
and then added a new method
template <typename Json> Json flatten_with_zero_padded_array_indexes(const Json &value) {
Json result;
std::basic_string<typename Json::char_type> parent_key;
flatten_(parent_key, value, result, true);
return result;
}
The original method calls with flatten_ with the bool parameter set to false
template <typename Json>
Json flatten(const Json& value)
{
Json result;
std::basic_string<typename Json::char_type> parent_key;
flatten_(parent_key, value, result, false);
return result;
}