param: optimize fix for exponential memory allocation

[0x2b3bfa0]

As @skshetry proposed on #10178 (comment), deduplicating key_paths would be enough, because dpath.search with distinct keys will never output the same object twice.

Important

This patch seems to fix #10177 and some other creative variants I wrote, but I'm still afraid of unforeseen edge cases; not sure if this is a good idea.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (abcd70b) 90.61% compared to head (ea173bb) 90.61%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10180   +/-   ##
=======================================
  Coverage   90.61%   90.61%           
=======================================
  Files         499      499           
  Lines       37881    37881           
  Branches     5505     5505           
=======================================
  Hits        34327    34327           
  Misses       2912     2912           
  Partials      642      642           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[skshetry]

As I said in #10178 (comment), this won't fix the issue because different keypaths could still reference to same list.

params:
- references
- references.references
- references.references.references
- references.references.references.references
- references.references.references.references.references
- references.references.references.references.references.references

I think there's no way around deepcopying. I'll suggest restoring that, and also making key_paths distinct (to reduce too much deep-copying).

[0x2b3bfa0]

As I said in #10178 (comment), this won't fix the issue because different keypaths could still reference to same list.

Do you have a minimal example for that? I have been unable to reproduce it, and can't think of any scenario where merge could use the same reference with all the keys being distinct. 🙈

[skshetry]

The childrens of the values that keypath/xpath points to could be same.

# dvc.yaml
stages:
  example:
    cmd: test
    params:
    - custom.yaml:
      - references
      - references.references
      - references.references.references
      - references.references.references.references
      - references.references.references.references.references
      - references.references.references.references.references.references

# custom.yaml
values: &anchor [0, 1, 2, 3]

references:
  references:
    references:
      references:
        references:
          references:
            references:
              first: *anchor
              second: *anchor
              third: *anchor
              fourth: *anchor
              fifth: *anchor
              sixth: *anchor
              seventh: *anchor
              eighth: *anchor
              ninth: *anchor
              tenth: *anchor
              eleventh: *anchor

[0x2b3bfa0]

Thanks! I didn't know that dpath.search had that behavior. 🤦🏼‍♂️

[skshetry]

We could try pruning sub-paths here, but with aliases and anchors, I cannot say with confidence if that would be enough. So, I think it's better to deepcopy and detach from shared references.