ds000248 aws cli download results in extra files

[larsoner]

Trying to find a workaround for #3695 by following the download instructions, I just ran:

$ pip install awscli
$ aws s3 sync --no-sign-request s3://openneuro.org/ds000248 ./ds000248 | tee output.txt
$ ls -al ./ds000248/derivatives/freesurfer/subjects
total 24
drwxrwxr-x 6 larsoner larsoner 4096 Jan 13 10:47 .
drwxrwxr-x 4 larsoner larsoner 4096 Jan 13 10:47 ..
drwxrwxr-x 8 larsoner larsoner 4096 Jan 13 10:47 01
drwxrwxr-x 9 larsoner larsoner 4096 Jan 13 10:47 fsaverage
drwxrwxr-x 2 larsoner larsoner 4096 Jan 13 10:48 morph-maps
drwxrwxr-x 9 larsoner larsoner 4096 Jan 13 10:48 sub-01
$ 

Looking at https://openneuro.org/datasets/ds000248/versions/1.2.4 I did not expect to see a derivatives/freesurfer/subjects/01 directory:

I noticed the error because having the 01 subdirectory in addition to the sub-01 subdirectory was problematic for some software I'm using.

It seems like a bug that this is included even though it's not in the latest dataset snapshot, but if this is expected behavior some docs in the download page would be useful (e.g., the wording "The most recently published snapshot can be downloaded from S3" should be adjusted)!

[pwighton]

I think I'm experiencing a similar issue with ds004230.

The web viewer shows two sessions per subject:

• ses-baseline
• ses-rescan

Which is what I was expecting, however the S3 bucket shows 4 session per subject:

bash-4.4$ aws s3 ls --no-sign-request s3://openneuro.org/ds004230/sub-PS11/
                           PRE ses-baseline/
                           PRE ses-baselinebrain/
                           PRE ses-rescan/
                           PRE ses-rescanbrain/

[effigies]

@pwighton @larsoner These two specific datasets should be fixed. Please let me know if they are not. We will leave this issue open until we have checked all datasets.

[larsoner]

Indeed I just tried my commands above and it's fixed, thanks @effigies !

[pwighton]

Thanks @effigies! I can also confirmed it's fixed in ds004230.