iterable_subprocess based annexworktree()

[mih]

This is moving PR mih#3 here, to get it tested properly. Below is the original description by @christian-monch

It includes (sits on top of) #538 (merged now)

TODO

• Rebase on Add a mode and test for non-recursive iter_gitworktree(), plus fixes #552
• Add documentation
• Make keep_ends=True str.strip() combo obsolete (and ease debugging why @mih cannot do it, ideally)
• fp=True currently only does meaningful things for annexed files, but it should also act properly for any other file. This aspect of the functionality is still undertested.

---

This PR adds an implementation of iter_annexworktree that is based on iterable_subprocesses and the ideas laid out in issue #537.

This PR included a collection of data processors that are basically generator-wrapper.

The PR also modifies iter_gitworktree to use iterable_subprocesses instead of the datalad-core runner.

The current implementation of iter_annexworktree iterates over a dataset with 33k annex files in less than 5 seconds on my machine.

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (8f404a1) 92.73% compared to head (8f27b7d) 92.87%.

Files	Patch %	Lines
datalad_next/iter_collections/annexworktree.py	98.48%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #539      +/-   ##
==========================================
+ Coverage   92.73%   92.87%   +0.14%     
==========================================
  Files         137      143       +6     
  Lines       10157    10365     +208     
  Branches     1103     1141      +38     
==========================================
+ Hits         9419     9627     +208     
  Misses        714      714              
  Partials       24       24              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mih]

After normalizing the benchmark conditions further, I have the following stats:

# this code
In [6]: %timeit list(iter_annexworktree('.'))
3.59 s ± 104 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# original sketch
In [7]: %timeit run_itersubproc()
4.17 s ± 317 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# baseline with iterable_subprocess
In [9]: %timeit list(iter_gitworktree('.'))
302 ms ± 8.02 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This compared to a shell pipe (using jq as consumer that added the JSON decoding load):

❯ multitime -n 5 git ls-files | git annex find --anything --format="\${key}\n" --batch | grep -v '^[[:space:]]*$' | git annex examinekey --json --batch | jq > /dev/null
===> multitime results
1: git ls-files
            Mean        Std.Dev.    Min         Median      Max
real        3.399       0.169       3.236       3.369       3.703       
user        0.006       0.005       0.000       0.003       0.012       
sys         0.006       0.006       0.000       0.008       0.015       

~8% difference, and nearly within the margin of error.

This works for me!

[mih]

#555 brings helpers that can be used to remove duplication of fp=True handling.

[mih]

This is done! A monumental effort. Thanks @christian-monch !