Add simple generate summaries and totals functions that group by directory.

[rwblair]

I used dataset in place of dandiset but that's not honest. It just so happens that all of Openneuro's datasets have their own root level prefix in the bucket. Because of that I'm not sure they're a good default.

Not worth merging as is, but more of a starting point to discuss what default behavior of update summaries and update totals should be.

[CodyCBakerPhD]

I used dataset in place of dandiset but that's not honest. It just so happens that all of Openneuro's datasets have their own root level prefix in the bucket. Because of that I'm not sure they're a good default.

Indeed I struggled with what to call these as well (even the phrase 'archive' is a bit close to our own use cases...)

The goal of the tool (though I need to make this clearer in some dev documentation on data structures...) is to create a mirror of the S3 bucket contents, where every object in the bucket is it's own directory with the 3 .txt files contained therein

The most general way I can think of referring to what we call data/dandisets is then, 'top level', as in 'top level summaries and totals' and we take that to mean datasets for OpenNeuro (because your top level is datasets) and DANDI of course has a special separate structure that has to be manually assembled (technically we even have other unmentionable things at our top-level besides blobs and zarr) to match the same concept

What do you think?

Not worth merging as is, but more of a starting point to discuss what default behavior of update summaries and update totals should be.

PR looks great! I will add more tests to enhance coverage in a follow-up, but I'd say this is good to merge once we get the 'mode' of extraction file patterns ironed out in the above comment

[CodyCBakerPhD]

LGTM @rwblair ready to merge?

I will add tests for all of this in a followup after the performance benchmarking