Unhandled exception in test-cbor.exe Stack overflow when debugging project with Visual Studio 2019 16.7.7 compiled with c++17 or c++latest

[Andreas-Schniertshauer]

In the file json.hpp in handle_value(Value&& v) an

Unhandled exception at 0x00007FF6D51C2CB8 in test-cbor.exe: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x0000004760A69FF8).

is thrown when debugging the test-cbor project in Visual Studio 2019 16.7.7 compiled with /std:c++17 or with /c++latest

This occures, because of recursive function calls: from_cbor, sax_parse, parse_cbor_internal, get_cbor_object, parse_cbor_internal, ...

Please describe the steps to reproduce the issue.

1. unzip json-3.9.1.zip
2. shell cd json-3.9.1
3. mkdir build
4. cd build
5. cmake .. -DJSON_BuildTests=On -D JSON_Install:BOOL=OFF -D JSON_ImplicitConversions:BOOL=OFF -DCMAKE_CXX_FLAGS="/std:c++17"
6. In test\download_test_data.vcxproj replace all cpp test -d json_test_data || with cpp if not exist json_test_data
7. Open nlohmann_json.sln in Visual Studio 2019 16.7.7
8. Right click test-cbor and select "Set as Startup Project" in the context menu
9. Switch configuration to Debug
10. Execute Build\Build solution (F7)
11. Execute Debug\Start Debugging (F5)

What is the expected behavior?

No stack overflow

And what is the actual behavior instead?

Stack overflow

Which compiler and operating system are you using?

• Compiler: Visual Studio 2019 16.7.7 Prof.
• Operating system: Windows 10 2004 Pro

Which version of the library did you use?

• latest release version 3.9.1 (zip file)

[nlohmann]

Maybe related: #1685

I cannot reproduce this issue in the CI. We also have OSS Fuzz running and there has not been a stack overflow reported in a long time. I have no idea what to make out of this issue.

[gregmarr]

Do you have a call stack showing line numbers of at least a full iteration of the infinite recursion?

[Andreas-Schniertshauer]

Yes, sure, here is the modified call stack, githubs comment box can only have 65536 chars, but the full call stack is 660090 chars:

Line numbers:
  8688 test-cbor.exe!nlohmann::detail::binary_reader<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,nlohmann::detail::iterator_input_adapter<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<unsigned char>>>>,nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::get_cbor_array(const unsigned __int64 len, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 8688  C++
  8733 test-cbor.exe!nlohmann::detail::binary_reader<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,nlohmann::detail::iterator_input_adapter<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<unsigned char>>>>,nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::get_cbor_object(const unsigned __int64 len, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 8733 C++
  8249 test-cbor.exe!nlohmann::detail::binary_reader<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,nlohmann::detail::iterator_input_adapter<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<unsigned char>>>>,nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::parse_cbor_internal(const bool get_char, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 8249  C++
  8303 test-cbor.exe!nlohmann::detail::binary_reader<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,nlohmann::detail::iterator_input_adapter<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<unsigned char>>>>,nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::parse_cbor_internal(const bool get_char, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 8303  C++

Call Stack:
  test-cbor.exe!std::vector<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,std::allocator<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::_Emplace_reallocate<enum nlohmann::detail::value_t>(nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>> * const _Whereptr, nlohmann::detail::value_t && <_Val_0>) Line 725 C++
  test-cbor.exe!std::vector<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,std::allocator<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::emplace_back<enum nlohmann::detail::value_t>(nlohmann::detail::value_t && <_Val_0>) Line 708  C++
  test-cbor.exe!nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>::handle_value<enum nlohmann::detail::value_t>(nlohmann::detail::value_t && v) Line 5480 C++
  test-cbor.exe!nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>::start_object(unsigned __int64 len) Line 5397 C++
  test-cbor.exe!nlohmann::detail::binary_reader<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,nlohmann::detail::iterator_input_adapter<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<unsigned char>>>>,nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::get_cbor_object(const unsigned __int64 len, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 8717 C++
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8733
  8303
  8688
  8249
  8733
  8303
  8688
  8249
  8688
  8249
  8733
  8303
  8733
  8303
  test-cbor.exe!nlohmann::detail::binary_reader<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>,nlohmann::detail::iterator_input_adapter<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<unsigned char>>>>,nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>>>::sax_parse(const nlohmann::detail::input_format_t format, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>> * sax_, const bool strict, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 7748 C++
  test-cbor.exe!nlohmann::basic_json<std::map,std::vector,std::string,bool,__int64,unsigned __int64,double,std::allocator,nlohmann::adl_serializer,std::vector<unsigned char,std::allocator<unsigned char>>>::from_cbor<std::vector<unsigned char,std::allocator<unsigned char>> &>(std::vector<unsigned char,std::allocator<unsigned char>> & i, const bool strict, const bool allow_exceptions, const nlohmann::detail::cbor_tag_handler_t tag_handler) Line 23968  C++
  test-cbor.exe!_DOCTEST_ANON_FUNC_208() Line 1926  C++
  test-cbor.exe!doctest::Context::run() Line 5864 C++
  test-cbor.exe!main(int argc, char * * argv) Line 5947 C++

[Andreas-Schniertshauer]

The file that is read is: build/json_test_data/json_testsuite/sample.json

[Andreas-Schniertshauer]

The last entry (465) of the variable ref_stack (size is 466) in function handle_value is:

[gregmarr]

What file is being passed to from_cbor? If it's really build/json_test_data/json_testsuite/sample.json then that's not right, as that's trying to read a JSON file as CBOR.

If it is being passed build/json_test_data/json_testsuite/sample.json.cbor then it could just be that the debug version isn't optimized enough and this deeply-nested file is just blowing up trying to read it. Increasing the stack size or running the release version instead of the debug version could help.

[gregmarr]

This does sound like the issue that @nlohmann referenced that was solved by increasing the stack size from the default 1MB to 8MB. #1685 (comment)

[Andreas-Schniertshauer]

What file is being passed to from_cbor? If it's really build/json_test_data/json_testsuite/sample.json then that's not right, as that's trying to read a JSON file as CBOR.

No, sorry I only looked for filename in unit-cbor.cpp and in line 1917 I found:

std::string filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json";

But a few lines later (1924) there is a .bor appended, so it is: build/json_test_data/json_testsuite/sample.json.cbor

[Andreas-Schniertshauer]

I solved the problem temporary by increasing the Stack Reserve Size.
I added in the foreach(file ${files}) loop of test\CMakeLists.txt:

    if (${testcase} STREQUAL "test-cbor")
        set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /STACK:4194304")
    endif()

This increases the stack size to 4MB and the test-cbor project can be debbuged in the Visual Studio debugger.

The problems are the recursive function calls, when the test data are increased the test may fail again, because then 4MB is not enough. The stack is finite and recursion uses the stack, so there is a fundamental limit on the number of calls that can be made.
The best solution would be to not use recursion at all.

[nlohmann]

The problem is conceptual, and I would be happy about PRs that propose a non-recursive implementation.

[gregmarr]

Did you try it in a release build? I imagine the stack usage is a lot less there due to inlining and other optimizations. If that's the case, I don't see this as a significant issue unless it's normal to have this level of nesting in production usage and the issue happening in release builds.

[Andreas-Schniertshauer]

This does sound like the issue that @nlohmann referenced that was solved by increasing the stack size from the default 1MB to 8MB. #1685 (comment)

Yes this seems to be the same problem, stack size too small.