Using json merge_patch on ordered non-alphanumeric datasets

[jjhand21]

• Describe what you want to achieve.
  I would like to use json merge patching in the following manner:
  1. Start with a default ordered dataset. The following is generic, but shows my intended formatting. Arrays are used to keep data from being reordered.

{
  "Main_Configuration": [
    {"First_Dataset" : [
      {"data_pts" : {
        "1" : [1, 3, 5, 7, 9],
        "2" : [60, 50, 40]
      } },
      {"names" : {
        "1" : "Paul",
        "2" : "John",
        "3" : "George"
      } }
    ]}, 
    {"Second_Dataset" : [
      {"Switch_values" : {
        "1" : true,
        "2" : false,
        "3" : false
      } },
      {"ratings" : [ 50, 70, 40, 90 ]},
      {"food" : "turkey"}
    ]}
  ]
}

2. A user adds value changes in a file with similar syntax:

{
  "Main_Configuration": [
    {"First_Dataset" : [
      {"data_pts" : {
        "3" : [3, 4, 5]        
      } },
    ]}, 
    {"Second_Dataset" : [
      {"food" : "pizza"}
    ]}
  ]
}

3. The patch would just insert the changes from the user file into the main file. Other settings inside objects or organization arrays would be untouched.

• Describe what you tried.
  I have tried the merge_patch method but it removes anything inside the high level organizational array of the main file and then just inserts what is in the patch. I imagine that removing the [ ] around the sets would allow this to work but I would lose the starting order of entries in my main file's input. This file will be dumped to the user to show them what settings were used during use of a software algorithm that uses the values so keeping the order the same is very helpful. I am attempting to see if "diff" will help in this situation, but if it does not I will have to find a way to have ordered data that can be patched in this manner so the user can have a complete set of ordered settings between defaults and their selected changes.

• Describe which system (OS, compiler) you are using.
  Redhat 7.6, GCC 4.8.5

• Describe which version of the library you are using (release version, develop branch).
  Release Version

[nlohmann]

I am not sure whether this is possible with JSON Merge Patch (https://tools.ietf.org/html/rfc7386). With JSON Patch (https://tools.ietf.org/html/rfc6902), however, it should be possible with an add operation.

[jjhand21]

Thanks for the reply. Yes, the merge_patch only seems to replace particular values if inside objects from the top level down because of the way that the algorithm works. I think that arrays may be treated just as a lowest level value which is just cleared and replaced with the merge data instead of leaving values that do not get changed in the new data.
I am trying to use json::diff(Main_Dataset, User_Changes) which creates the 'add' operation as you mentioned, yet it also adds 'remove' operations for the other sections of the main dataset. This achieves a similar output to the merge_patch function. Is there any way to just keep the add operations via other patch operations or will I have to edit the diff output?

[nlohmann]

By the way: the update function may also be helpful.

[jjhand21]

I have found through testing that "update" also seems to clear anything in the highest level array and write in only the changes instead of adding them to other existing information. It appears to share the same overall design ideas of "patch" and "merge_patch". I'm sure that this is an unintended flaw since many json users do not care about the ordering of the data keys in the output. I just need to maintain that ordering for the users who look at the settings after using our software so I use arrays to ensure the order stays the same.

On the other hand, I was able to manually create json_pointer lines that add new values to existing data or overwrite the previous values correctly by creating a unicode string for the locations. I would have to find the locations ahead of every merge with specific 'find' operations and somehow use the json_pointer syntax. I tried substituting a generated string into the syntax but it does not seem to work, for example:

std::string path = "/Main_Configuration/0/First_Dataset/0/data_pts/2";
std::vector temp = [3, 4, 5];
MainDatafile[(path)_json_pointer] = temp;     // Fails because of "(path)"

Is there a way to use automatically generated strings in the pointer syntax for my update problem? Also, has any users had luck doing 'update' style changes to their json if they use the "fifo_map" format that you have suggested in other posts for ordered data?

[nlohmann]

You can create a JSON Pointer from a string with this constructor: https://nlohmann.github.io/json/classnlohmann_1_1json__pointer_a7f32d7c62841f0c4a6784cf741a6e4f8.html#a7f32d7c62841f0c4a6784cf741a6e4f8

[jjhand21]

Ok, thanks for the link.

[nlohmann]

@jjhand21 Do you need further assistance on this issue?

[jjhand21]

No. I decided to use pyhocon on the data ingest in order to meet other requirements I had, and then merge in that format before handling data as a pure json type. Thanks for the suggestions though.