Removing duplicates while transforming flat nodes to hierarchy.

[pratikpparikh]

@larsga I have another unique problem in dealing with nested structures and need your opinion. I have duplicate reference node, I need to figure out how to private the duplicate from being created. I started building a global array and thinking that I need to check that array in flattening to remove duplicates. If you can give it a look and let me know what you think the best option here is then it will be great. Below is current input, expected output, and the current template.

Input

[
{
"chapters": [
{
"name": "a-f"
},
{
"name": "b"
}
]
},
{
"chapters": [
{
"name": "a-f"
},
{
"name": "e"
}
]
},
{
"chapters": [
{
"name": "l-p"
},
{
"name": "m"
}
]
},
{
"chapters": [
{
"name": "v-z"
},
{
"name": "v"
}
]
},
{
"chapters": [
{
"name": "v-z"
},
{
"name": "z"
}
]
}
]

Expected Output

[
{
"name": "a-f"
},
{
"name": "b",
"parents": [
{
"name": "a-f"
}
]
},
{
"name": "e",
"parents": [
{
"name": "a-f"
}
]
},
{
"name": "l-p"
},
{
"name": "m",
"parents": [
{
"name": "l-p"
}
]
},
{
"name": "v-z"
},
{
"name": "v",
"parents": [
{
"name": "v-z"
}
]
},
{
"name": "z",
"parents": [
{
"name": "v-z"
}
]
}
]

Current Template

def flatten(arrayofarrays)
if ($arrayofarrays)
$arrayofarrays[0] + flatten($arrayofarrays[1 : ])
else
[]

let globalTopNodeCache = [for (.) if(.chapters and size(.chapters) > 0) .chapters[0].name]

flatten([for (.)
let chapterZero = if(.chapters and size(.chapters) > 0) .chapters[0] else ""
[{"name" : $chapterZero .name}] + [for (.chapters[1 : ]) {"parents":[{"name":$chapterZero .name}],"name" : .name}]
])

[larsga]

This gets pretty close:

def unique(values)
  let obj = {for ($values) . : .}
  [for ($obj) .value]

let chapters = [for (.) .chapters]
let parents = [for (unique([for ($chapters) .[0].name])) {"name" : . }]
let children = [for ($chapters) {
  "name" : .[1].name,
  "parents" : {"name" : .[0].name}
}]

$parents + $children

The only thing is that here the order is different, because the parents come first, instead of being interleaved with the children.

The interleaving can be done by nesting and then using flatten, but it requires supporting if in forloops. Which is the same thing we needed in the previous issue.

def flatten(arrayofarrays)
  if ($arrayofarrays)
    $arrayofarrays[0] + flatten($arrayofarrays[1 : ])
  else
    []

def unique(values)
  let obj = {for ($values) . : .}
  [for ($obj) .value]

let input = (.)
let chapters = unique([for (.) .chapters[0].name])

flatten([for ($chapters) 
    let name = (.)
   
    [
    {"name" : .},
    [for ($input) if (.chapters[0].name == $name) {
      "name" : .chapters[1].name,
      "parents" : {"name" : .chapters[0].name}
    }]
  ]
])

[pratikpparikh]

@larsga , Can you explain to me how the unique is being identified?

def unique(values)
let obj = {for ($values) . : .} // is this creating set?
[for ($obj) .value]

[larsga]

Yes, we create a JSON object on the second line where the values are used as the key. This causes duplicate keys to overwrite each other, so that when we extract them in the third line we get only the unique values.

I've thought a bit more about this, and probably group by is the best long-term solution. That would work as follows: you can add group by <expr> to a for loop. This will cause your for loop to loop over the following structure:

[
  {"key" : key1,
   "values" : obj1, obj2, obj3, ... },

  {"key" : key2,
   "values" : obj4, obj5, obj6, ... },

  ...
]

In other words, you'll loop over an array of objects where each object has the group by-value (key) and the set of values from the original for (<expr>) that produced that exact key.

With that the implementation becomes:

flatten([for (.)
  let key = .key
  [
    {"name" : $key},
    [for (.values) {
      "name" : .[1].name,
      "parents" : {"name" : $key}
    }]
  ]
group by .chapters[0].name])

[pratikpparikh]

@larsga can we take a small example and play this out. I understand the concept, but it would much better if we can clearly visualize the states of transformation as both if and group by are not yet supported. I will give this a try, please correct me as I tumble :

stage 0: not sure how we got here????
[
{"key" : "a-f",
"values" : ["b", "e"] },

{"key" : "l-p",
"values" : ["m"] },
...
]

stage 1: we get a structure intended of parent first and children subarray.
[
{
"name": "a-f"
},
[
{
"name": "b",
"parents" : [{
"name" : "a-f"
}]
},
{
"name": "e",
"parents" : [{
"name" : "a-f"
}]
}
]
]

stage 2: just flatten.

[larsga]

Stage 0: that's what the group by does. Simply including the group by in the for expression makes you loop over that structure. That's basically how group by works. Maybe the idea writeup will help.

But the values are not as you show them. Instead they are

{"key" : "a-f",
  "values" : [

{
"chapters": [
{
"name": "a-f"
},
{
"name": "b"
}
]
},
{
"chapters": [
{
"name": "a-f"
},
{
"name": "e"
}

]}

group by doesn't know which part of the object you want, so it keeps everything.

Stage 1 is correct, and I think it's clear how it happens.

[pratikpparikh]

@larsga am on the same page now. How hard is it to implement group by and if support for for-loop?

[larsga]

If support for the for loop is implemented now. So you have a solution now.

Group by is easy to implement, but I want to research alternatives a little more before implementing it.

[larsga]

Some notes here.

[pratikpparikh]

@larsga yes, thank you, I have been tracking the changes. Have not fully consumed group by and order by thought process but will provide feedback as soon as I do.

[pratikpparikh]

@larsga very well documented sir. Thank you.

Q- What if I want to group by multiple items?
A- We will just have to create a key with object (multiple name-value), it should be simple but still asking so we can clearly document.

1. I like xQuery, very easy on the eyes.
2. I love comprehensions, but users now have to learn the ordering which can prevent correct utilization.
3. macro seems to be a good compromise. So, I think my vote is to that at this point.

If i understanding everything correctly, I also think macro will provide us with more flexibility of implementation as you will only be introducing built-in function rather than language. Please let me know what you think? I also think we need to use what is best for performance.

[pratikpparikh]

@larsga Hope you had a great vacation and I did not take up too much of your time during your time off. If you get time can you look at group-by, I tried to give it a shot by creating a macro but don't feel it works as intended because it does grouping only by value and not path and value? So if you can look at this when you are back into the groove then it will be nice of you.

[zsyphon]

@larsga @pratikpparikh HI guys I've run into a similar group by issue as well. Have you been able to make any progress on this? Thank you for any help or updates you can provide!