Add support for new MNE-BIDS BIDSPath #146

hoechenberger · 2020-07-01T13:30:59Z

No description provided.

03-make_epochs.py

04a-run_ica.py

hoechenberger · 2020-07-02T08:45:20Z

The somato dataset fails to download on Circle

hoechenberger · 2020-07-02T11:52:39Z

The somato dataset fails to download on Circle

Opened an issue upstream:
https://github.com/sappelhoff/MNE-somato-data/issues/1

sappelhoff · 2020-07-02T16:01:51Z

The somato dataset fails to download on Circle

@hoechenberger it should be fixed now. I also wrote a guide so that anyone can fix this problem on their own (click on "details").

These should be the needed steps. I am doing them now and will update this post if it turns out that one of the steps is wrong.

general requirements:

datalad
git annex

Get data as .tar.gz from MNE OSF and turn into datalad dataset

wget -q -O - https://osf.io/tp4sg/download?version=6 | tar xvzf -
mv MNE-somato-data ..
datalad create MNE-somato-data
cd MNE-somato-data
mv ../../MNE-somato-data/* .
download fsaverage.zip from https://osf.io/rxvq7/, unpack it, and use the resulting fsaverage directory to replace the symlink MNE-somato-data/derivatives/freesurfer/subjects/fsaverage
datalad save
rm -rf ../../MNE-somato-data

Upload file by file to a new OSF repo

using https://github.com/datalad/datalad-osf

export OSF_TOKEN=<your token created from OSF ui>
datalad create-sibling-osf -s osf MNE-somato-data --mode export
git annex export HEAD --to osf

... this may take a long time depending on your upload speed.

... it may happen that not all files get uploaded (due to connection error). In that case, see: datalad/datalad-osf#109

Parse the new OSF, download urls into a CSV and force update a dataset (the GitHub mirror) ... or make a new GitHub mirror

using https://github.com/templateflow/datalad-osf ... note that you probably have to remove datalad/datalad-osf from your namespace first
here, we update an existing mirror ... but for making a new mirror, see gsoc2019 blog
more info on updating partials of a datasets neurostars issue

make the new OSF repo public
cd ..
mkdir gh
cd gh
datalad clone https://github.com/sappelhoff/MNE-somato-data
cd MNE-somato-data
python -c "import datalad_osf; datalad_osf.update_recursive(key='nxtqh', csv='./tmp.csv', limit_to_ext=None)" ... replace nxtqh with your OSF key (the suffix of the URL)
rm tmp.csv
datalad save
datalad publish

02-frequency_filter.py

jasmainak

I didn't look too closely for typos etc but PR looks good to me otherwise. The copy are a bit awkward but I guess there is no way around it.

sappelhoff · 2020-07-02T16:39:28Z

the somato thing still fails, because the fsaverage symlink was not uploaded ... and OSF refuses to accept it, because it thinks it's a directory ...

no luck with osf-client or datalad-osf either. No clue how I managed to do that back then ...

sappelhoff · 2020-07-02T17:05:21Z

no luck with osf-client or datalad-osf either. No clue how I managed to do that back then ...

okay, found the issue. I currently don't have freesurfer installled ... so the symlink is broken on my system. and apparently, broken symlinks behave very odd.

hoechenberger · 2020-07-02T17:08:28Z

Thanks for looking into this, @sappelhoff! Can you ping me when you think it's ready to go? Just tried fetching somato and still hitting the same error :(

sappelhoff · 2020-07-02T17:21:22Z

Thanks for looking into this, @sappelhoff! Can you ping me when you think it's ready to go? Just tried fetching somato and still hitting the same error :(

yeah, the plot thickens. I now managed to upload a fsaverage symlink to OSF and also managed to push it to the GitHub mirror ...

now the only question is whether that will work. Because I think beforehand, the git-annex data contained the actual contents of fsaverage, instead of a symlink.

These contents are:

label mri mri.2mm scripts surf xhemi

apparently I copied them from the symlink's destination /usr/local/freesurfer/subjects/fsaverage to the fsaverage in the MNE-somato dataset.

The odd thing is, that the MNE .tar.gz version contains a symlink.

Well ... I just hope it works now. Can you try again?

Else I'll have to ask someone to share these files from their freesurfer installation folder "label mri mri.2mm scripts surf xhemi" and I'll upload them.

tests/download_test_data.py

sappelhoff · 2020-07-02T17:42:03Z

nice, so now we don't have a download error any more ... the symlink seems to work somewhat.

We get a new error though:

17:39:25  A critical error occurred.
 
Traceback (most recent call last):
 
  File "tests/run_tests.py", line 138, in <module>
 
    run_tests(test_suite)
 
  File "tests/run_tests.py", line 106, in run_tests
 
    pipeline()
 
  File "tests/run_tests.py", line 48, in source
 
    mod.main()
 
  File "/home/circleci/project/10-make_forward.py", line 83, in main
 
    itertools.product(config.get_subjects(), config.get_sessions()))
 
  File "/home/circleci/project/10-make_forward.py", line 82, in <genexpr>
 
    parallel(run_func(subject, session) for subject, session in
 
  File "/home/circleci/project/config.py", line 1046, in wrapper
 
    raise(e)
 
  File "/home/circleci/project/config.py", line 1039, in wrapper
 
    return func(*args, **kwargs)
 
  File "/home/circleci/project/10-make_forward.py", line 68, in run_forward
 
    subjects_dir=config.get_fs_subjects_dir())
 
  File "<decorator-gen-72>", line 21, in make_bem_model
 
  File "/home/circleci/miniconda/lib/python3.7/site-packages/mne/bem.py", line 590, in make_bem_model
 
    surfaces = _surfaces_to_bem(surfaces, ids, conductivity, ico)
 
  File "/home/circleci/miniconda/lib/python3.7/site-packages/mne/bem.py", line 500, in _surfaces_to_bem
 
    surfs[si] = read_surface(surf, return_dict=True)[-1]
 
  File "<decorator-gen-66>", line 21, in read_surface
 
  File "/home/circleci/miniconda/lib/python3.7/site-packages/mne/surface.py", line 720, in read_surface
 
    ret = _get_read_geometry()(fname, read_metadata=read_metadata)
 
  File "/home/circleci/miniconda/lib/python3.7/site-packages/nibabel/freesurfer/io.py", line 183, in read_geometry
 
    raise ValueError("File does not appear to be a Freesurfer surface")
 
ValueError: File does not appear to be a Freesurfer surface

hoechenberger · 2020-07-02T17:51:48Z

nice, so now we don't have a download error any more ... the symlink seems to work somewhat.

I'm getting the following error message:

(mne-dev) ➜  tests git:(bidspath) python download_test_data.py -d somato

----------------------
datalad installing "somato"
[INFO   ] Cloning https://github.com/sappelhoff/MNE-somato-data to '/home/parietal/rhochenb/mne_data/somato' 
datalad get data "sub-01" for "somato"
[INFO   ] Actually getting 8 files 
datalad get data "derivatives/freesurfer/subjects" for "somato"                                                  
[INFO   ] Actually getting 504 files                                                                             
Total (503 ok, 1 failed out of 504): 100%|████████████████████████████████████| 345M/345M [01:08<00:00, 12.0MB/s][WARNING] Running get resulted in stderr output:   download failed: Gone                                         
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone                                                                                          
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download failed: Gone
  download faile... 
                                                                                                                 [ERROR  ] from web...
| Unable to access these remotes: web
| Try making some of these repositories available:
|       00000000-0000-0000-0000-000000000001 -- web
|       df1ecd42-2c7f-4e08-ad85-e4255e20fb1b -- stefanappelhoff@arc-lin-004309:~/Desktop/bids/MNE-somato-data
| 
| (Note that these git remotes have annex-ignore set: origin) [get(/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/fsaverage)] 
[WARNING] could not get some content in /home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects ['/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/fsaverage'] [get(/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects)] 
Traceback (most recent call last):
  File "download_test_data.py", line 104, in <module>
    main(dataset)
  File "download_test_data.py", line 93, in main
    dataset.get(to_get, jobs=n_jobs)
  File "/storage/store/work/rhochenb/miniconda3/envs/mne-dev/lib/python3.8/site-packages/datalad/distribution/dataset.py", line 437, in apply_func
    return f(**kwargs)
  File "/storage/store/work/rhochenb/miniconda3/envs/mne-dev/lib/python3.8/site-packages/datalad/interface/utils.py", line 439, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/storage/store/work/rhochenb/miniconda3/envs/mne-dev/lib/python3.8/site-packages/datalad/interface/utils.py", line 427, in return_func
    results = list(results)
  File "/storage/store/work/rhochenb/miniconda3/envs/mne-dev/lib/python3.8/site-packages/datalad/interface/utils.py", line 413, in generator_func
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully [{'type': 'file', 'refds': '/home/parietal/rhochenb/mne_data/somato', 'status': 'error', 'path': '/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/fsaverage', 'action': 'get', 'annexkey': 'MD5E-s0--d41d8cd98f00b204e9800998ecf8427e', 'message': 'from web...\nUnable to access these remotes: web\nTry making some of these repositories available:\n\t00000000-0000-0000-0000-000000000001 -- web\n \tdf1ecd42-2c7f-4e08-ad85-e4255e20fb1b -- stefanappelhoff@arc-lin-004309:~/Desktop/bids/MNE-somato-data\n\n(Note that these git remotes have annex-ignore set: origin)'}, {'action': 'get', 'path': '/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects', 'type': 'directory', 'refds': '/home/parietal/rhochenb/mne_data/somato', 'status': 'impossible', 'message': ('could not get some content in %s %s', '/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects', ['/home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/fsaverage'])}]
(mne-dev) ➜  tests git:(bidspath)

sappelhoff · 2020-07-02T17:55:33Z

it's weird that you get a different error than the CI.

but anyhow, I am now downloading the full fsaverage data, then uploading to OSF, then updating the datalad mirror.

Then everything should be as it was before. 😬

hoechenberger · 2020-07-02T18:34:59Z

it's weird that you get a different error than the CI.

Yeah idk… I deleted the entire somato directory before retrying. Maybe the CI's cache is playing tricks on us here.

sappelhoff · 2020-07-02T20:42:12Z

Can you try again @hoechenberger? I re-did the MNE-somato-data repository. It should be exactly as it was before now.

hoechenberger · 2020-07-02T20:46:50Z

Can you try again @hoechenberger? I re-did the MNE-somato-data repository. It should be exactly as it was before now.

It's working for me locally! Thanks for your help, @sappelhoff! I will restart CI now.

hoechenberger · 2020-07-02T21:11:05Z

@sappelhoff Sorry, this was a bit premature. I only checked whether the download of the dataset works properly now, which it does (at least doesn't throw an error message). But locally I also get ValueError: File does not appear to be a Freesurfer surface. Will try to see if I can figure out where the problem is coming from. But I would bet it's related to that symlink thingy you mentioned before…

hoechenberger · 2020-07-02T21:30:20Z

@sappelhoff There is indeed still sth wrong with the link:

$ ls -lH /home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/01/bem/inner_skull.surf
-r--r--r-- 1 rhochenb parietal 32 Jul  2 22:44 /home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/01/bem/inner_skull.surf
$ cat /home/parietal/rhochenb/mne_data/somato/derivatives/freesurfer/subjects/01/bem/inner_skull.surf
watershed/01_inner_skull_surface%
$

sappelhoff · 2020-07-03T07:47:59Z

this is an absolute nightmare, I hope this gets easier with the great work that @mih is doing in datalad/datalad-osf#106

agramfort · 2020-07-03T08:14:11Z

we can also put the dataset on openneuro so we have all dataset on openneuro. https://github.com/mne-tools/mne-study-template/blob/master/tests/download_test_data.py#L17 somato is the only one on OSF

hoechenberger · 2020-07-03T09:17:48Z

we can also put the dataset on openneuro so we have all dataset on openneuro.

https://github.com/mne-tools/mne-study-template/blob/master/tests/download_test_data.py#L17

somato is the only one on OSF

+1!!

sappelhoff · 2020-07-03T09:37:36Z

I am trying one last thing right now (uploading the fsaverage/bem folder as well), if that doesn't work, I am at my wit's end anyhow.

So the only options would be:

upload to OpenNeuro
Wait for datalad/datalad-osf/106

sappelhoff · 2020-07-03T10:18:14Z

okay, that didn't work either. I give up for now.

agramfort · 2020-07-03T10:22:13Z

openneuro it is then !

hoechenberger · 2020-07-03T11:39:02Z

Let me know if I can be of any help here!

sappelhoff · 2020-07-03T12:52:28Z

Let me know if I can be of any help here!

yes, can you upload what you need to OpenNeuro, please? :)

I already broke the dataset, so I feel my job is done /sarcasm

but for real though, fighting with datalad yesterday and today already cost me several hours, so it'd be nice if you could take over 👍

hoechenberger · 2020-07-03T12:59:12Z

Ok I will see what I can do, @sappelhoff! Just need to find a source that's not broken ;) Will keep you posted! Thanks for putting so much time & effort into this, I really appreciate it.

hoechenberger · 2020-07-03T14:11:52Z

I've published it on OpenNeuro: https://openneuro.org/datasets/ds002993/

However, it's yet to be synced to GitHub: https://github.com/OpenNeuroDatasets/ds002993

Anyone know how long this usually takes?

hoechenberger · 2020-07-03T14:12:59Z

However, it's yet to be synced to GitHub: https://github.com/OpenNeuroDatasets/ds002993

Heh, it just finished syncing! Neat!

hoechenberger · 2020-07-03T14:47:24Z

Yeeeeesssss!! It's working!!!
@agramfort care to merge? :)

sappelhoff · 2020-07-03T15:11:46Z

nice, and interestingly without fsaverage link whatsoever.

I am going ahead and will delete https://github.com/sappelhoff/MNE-somato-data and the corresponding OSF repo.

Thanks for cleaning up @hoechenberger!

hoechenberger · 2020-07-03T15:14:45Z

nice, and interestingly without fsaverage link whatsoever.

I intentionally removed it – was that a bad idea?

sappelhoff · 2020-07-03T16:36:39Z

I intentionally removed it – was that a bad idea?

if it works, it must be fine. The fsaverage is shipped with freesurfer anyways, so it was a mystery to me anyhow why I previously included it in the dataset before 🤷‍♂️

hoechenberger · 2020-07-03T16:38:12Z

The fsaverage is shipped with freesurfer anyways, so it was a mystery to me anyhow why I previously included it in the dataset before

This was my thinking exactly!

jasmainak · 2020-07-03T16:51:32Z

Is this somato data also used in MNE? Something to update there?

hoechenberger · 2020-07-03T17:38:15Z

Is this somato data also used in MNE? Something to update there?

To my knowledge it's the exact same dataset. MNE should probably start fetching it from OpenNeuro too now, yes :)

sappelhoff · 2020-07-03T17:50:00Z

Is this somato data also used in MNE? Something to update there?

yup, MNE uses the BIDS version of the somato data, as stored on OSF

in fact, I don't think there is a non-bids version anymore (except of course in the OSF version history)

MNE should probably start fetching it from OpenNeuro too now, yes :)

xref: mne-tools/mne-python#6687

:)

hoechenberger · 2020-07-03T21:22:09Z

Thanks @agramfort!

Add support for new MNE-BIDS BIDSPath

e68fffc

jasmainak reviewed Jul 1, 2020

View reviewed changes

03-make_epochs.py Outdated Show resolved Hide resolved

jasmainak reviewed Jul 1, 2020

View reviewed changes

04a-run_ica.py Outdated Show resolved Hide resolved

hoechenberger added 2 commits July 2, 2020 09:28

Correct filenames

1d9939f

Simplify

3633830

Add bids_root as prefix via make_bids_basename

a057d87

hoechenberger mentioned this pull request Jul 2, 2020

should BIDSPath include bids_root mne-tools/mne-bids#454

Closed

hoechenberger added 2 commits July 2, 2020 13:01

Fix path

88b4366

Fix emptyroom path

35f05d4

hoechenberger requested a review from jasmainak July 2, 2020 11:53

jasmainak reviewed Jul 2, 2020

View reviewed changes

02-frequency_filter.py Show resolved Hide resolved

jasmainak approved these changes Jul 2, 2020

View reviewed changes

tmp patch for fsaverage symlink in somato

339044a

sappelhoff reviewed Jul 2, 2020

View reviewed changes

tests/download_test_data.py Outdated Show resolved Hide resolved

remove patch, not working, problem is different than thought

228b243

Retrieve somato dataset from OpenNeuro

c0f0d50

agramfort approved these changes Jul 3, 2020

View reviewed changes

agramfort merged commit fd3bfd5 into mne-tools:master Jul 3, 2020

hoechenberger deleted the bidspath branch July 3, 2020 21:22

Add support for new MNE-BIDS BIDSPath #146

Add support for new MNE-BIDS BIDSPath #146

Conversation

hoechenberger commented Jul 1, 2020

hoechenberger commented Jul 2, 2020

hoechenberger commented Jul 2, 2020

sappelhoff commented Jul 2, 2020 • edited Loading

Get data as .tar.gz from MNE OSF and turn into datalad dataset

Upload file by file to a new OSF repo

Parse the new OSF, download urls into a CSV and force update a dataset (the GitHub mirror) ... or make a new GitHub mirror

jasmainak left a comment

Choose a reason for hiding this comment

sappelhoff commented Jul 2, 2020

sappelhoff commented Jul 2, 2020

hoechenberger commented Jul 2, 2020

sappelhoff commented Jul 2, 2020

sappelhoff commented Jul 2, 2020

hoechenberger commented Jul 2, 2020 • edited Loading

sappelhoff commented Jul 2, 2020

hoechenberger commented Jul 2, 2020

sappelhoff commented Jul 2, 2020

hoechenberger commented Jul 2, 2020

hoechenberger commented Jul 2, 2020

hoechenberger commented Jul 2, 2020

sappelhoff commented Jul 3, 2020

agramfort commented Jul 3, 2020 via email

hoechenberger commented Jul 3, 2020

sappelhoff commented Jul 3, 2020

sappelhoff commented Jul 3, 2020

agramfort commented Jul 3, 2020 via email

hoechenberger commented Jul 3, 2020

sappelhoff commented Jul 3, 2020

hoechenberger commented Jul 3, 2020 • edited Loading

hoechenberger commented Jul 3, 2020

hoechenberger commented Jul 3, 2020

hoechenberger commented Jul 3, 2020

sappelhoff commented Jul 3, 2020

hoechenberger commented Jul 3, 2020

sappelhoff commented Jul 3, 2020

hoechenberger commented Jul 3, 2020

jasmainak commented Jul 3, 2020

hoechenberger commented Jul 3, 2020

sappelhoff commented Jul 3, 2020

hoechenberger commented Jul 3, 2020

sappelhoff commented Jul 2, 2020 •

edited

Loading

hoechenberger commented Jul 2, 2020 •

edited

Loading

hoechenberger commented Jul 3, 2020 •

edited

Loading