ENH: Adds populate_intended_for for fmaps #482
Conversation
It also adds the corresponding unit test
It adds a function to populate the "IntendedFor" field to the fmap jsons. It also adds the corresponding test. Minor modification to `create_tree` for the case of a `name` ending in `.json`.
It also adds the corresponding test
Also, fixes minor bug in ignoring `_sbref` images.
...rather than underscores (`_`) for consistency with other commands
For the case magnitude/phase fmaps.
Merges nipy's master
pvelasco
force-pushed
the
adds_populate_intended_for
branch
from
2699959
to
34b2acc
Compare
Codecov Report
@@ Coverage Diff @@
## master #482 +/- ##
==========================================
+ Coverage 77.89% 81.20% +3.30%
==========================================
Files 41 41
Lines 3203 3782 +579
==========================================
+ Hits 2495 3071 +576
- Misses 708 711 +3
Continue to review full report at Codecov.
|
It also adds the corresponding unit tests
No change in functionality
|
THANK YOU @pvelasco , I will review/try it out in upcoming days and provide feedback. Happy holidays! |
…s expected IntendedFor This way, when we write future test cases, the test function can be reused as long as it know what the expected IntendedFor are.
Removing an element from a (sorted) list while looping through the list was not a good idea, and it failed in some cases. Now we keep a parallel list with elements already accounted for and, while looping through the list, we only act if the element is not already accounted for.
It coves the cases in which the fmaps are in the form of magnitude/phase maps
To prepare for a future `test_convert.test_convert`, which requires mocking some functions
It also adds the corresponding unit tests.
We make sure we don't try to write to the root directory
|
I saw activity ongoing through the productive (for some) holidays thus did not review. Is it ready @pvelasco ? ;-) |
|
Yes. I think it is ready. I had some bugs in the unit tests, since I could not test them locally (#486), but they are now fixed. |
There was a problem hiding this comment.
I only started to review (so review is not complete, but decided to not delay until it is "well done" ;)), and still thinking about it
- I really like an idea to (ab)use
_acq-as an indicator for such grouping. I even at some point thought to establish a dedicated one withinreproinheuristic, so the "loosely specified in BIDS"_acq-seems to be reasonable one to use. I just still wonder if such new behavior would not somehow introduce some undesired assignment, in particular if ShimSettings are not there (Philips, GE, ?). - we might want to add additional, or alternative (if no ShimSettings?) constraints, e.g. based on resolution and "sform" (have not looked what would be the analog at DICOM level, ImageOrientationPatientDICOM?). If I got it right it is still debatable if fieldmaps could be acquired/reconstructed at resolution different from the target images... but may be we for starters could constrain based on entire sform, thus at least guaranteeing that positioning and sizes are the same
- in one of our use cases I noted that shimming was just a bit different:
[-7371, 5580, 11925, -269, -131, -448, -111, -459] vs
[-7372, 5589, 11937, -276, -130, -360, -157, -451]
yet to clarify with original collection site/techs -- but it raised the question on either it is completely useless, or we could still associate, and thus here may be allow for some %difference?
- in case of ambiguity (multiple fmaps matching) - should we take the one closest in time?
- since different people might want/have different strategies - should this be more of a heuristic, i.e. other heuristics could reuse in theirs etc... or at least may be we should use this one only if heuristic does not provide one
heudiconv/bids.py
Outdated
| j for j in glob(op.join(path_to_bids_session, '*/*.json')) if not ( | ||
| j in fmap_jsons | ||
| # j[:-5] removes the '.json' from the end | ||
| or j[:-5].endswith('_sbref') |
There was a problem hiding this comment.
just thinking out loud: we so need "BIDS filename" parser -- our code is full of similar constructs. #452
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
|
Hello @pvelasco and @yarikoptic , this is a great PR ! I had some issues when using it in my case. The main problems were:
I made a PR to address the first issue here, and i already have code to address the second issue but it'd be easier if the first PR could be validated before i submit the second PR. What do you guys think ? |
BF: In `find_compatible_fmaps_for_run`, actually check if the first element of the parameters list is a string. Thanks, @neurorepro
|
@pvelasco and @yarikoptic I have now made this PR to address custom labels, which includes updated doc and test suite. |
Add matching by custom label Thank you, @neurorepro
|
@pvelasco and @yarikoptic , how can i help finalize this PR so that it can be merged into the main branch ? |
| update_json(fm_json, {"IntendedFor": intended_for}, pretty=True) | ||
|
|
||
|
|
||
| class BIDSFile(object): |
There was a problem hiding this comment.
woohoo -- here it is also! ideally #544 should be finished/merged first (so would be driven by a specific copy of the schema etc), but doesn't need to be. If this PR is merged first, that one should be rebased/code moved/RFed and would immediately take advantage of this use-case use
clearly I would not be able to evaluate it in full/detailed, but I will resolve conflicts now and may be do some minor tune ups, and push back... then whenever green -- we should just merge and go from there. meanwhile -- more testing never hurts! |
…ded_for * origin/master: Update test to match new behavior. Drop suffix parameter from update_complex_name(). Remove unused imports. Enhance lists of entities in filename updaters. Use "base_fn" instead of "fn" in tests. Rename suffix to suffix_counter in update_complex_name. Work on fixing issues with multi-file converters. BF: Fix the order of the 'echo' entity in the filename try a simple fix for wrongly ordered files in tar file Use False instead of NaN bc filter wasn't catching NaNs. Convert variables to lists and check for lists in functions. Conflicts: heudiconv/tests/test_convert.py -- just in imports
|
Thank you @pvelasco for all the work on this PR and thank you @neurorepro for the help! I did some testing locally -- seems to work nicely. Let's proceed! |
|
Thanks for very useful addition @pvelasco, @neurorepro, and @yarikoptic. I am doing some local tests with the main branch, and seeing this warning when I run with "--command populate-intended-for" option. Interestingly, if I remove "--command populate-intended-for" option, as long as I have POPULATE_INTENDED_FOR_OPTS in my heuristics, the fmap jsons get populated correctly with "IntendedFor" field! |
|
Hi @nikhil153, The warning you receive when running with the As for the observation:
I can replicate the behavior and I'm looking into it. |
|
If there is an issue/question -- please file a new issue (may be referencing this PR). This PR was merged and should RiP ;) |
|
🚀 PR was released in |
|
Hi @pvelasco I meet the same issue that For example, So this slight difference will have an effect on This is also causing heudiconv errors when adding Best, |
|
Hi @WangYunHong98, Lately I switched jobs and now I'm working on different projects. I haven't been able to work on this project for about two years, but I think the way to make the parameter comparison less restrictive would be to add a relative tolerance in the call to That said, first principles say that if the Some people smarter than me argue that using an approximate field correction is better than no field correction but, afaik, there have been no publication examining whether that's true or not, or what can be considered "slightly different parameters". So I would argue that a very good solution would be to have a configuration parameter somewhere that allows you to specify how restrictive you want to be, and let the user specify it. Right now, I don't have the bandwidth to make those changes. Do you feel up to the task? 😉 |
This commits adds a new
populate_intended_forfunctionality toheudiconv. As its name indicates, it populates the"IntendedFor"field in thefmapjson files of a given session.At the moment, it can be called as a
--commandoption. In the future, it should also be called automatically every time there is a bids extraction."ShimSetting"field is present in thefmapjson file, it must match the"ShimSetting"in the other run (func,dwi, etc.) for this other run to be added to the"IntededFor"for thatfmap. If the"ShimSetting"is not present in the json files, the match is based on the<acq>entity in thefmapfile name: jsons withacq-dwiwill be intended for runs in thedwifolder,acq-fmrioracq-funcwill be intended for runs in thefuncfolder, etc. The logic for enforcing that the"ShimSetting"(if present) matches is that if the scanner changes the shims between afmapand its"IntededFor"run, the actual B0 field will be different between them, so the field measured (or estimated, from apepolarpair) from thefmapshould not be applied to correct the other run.fmapfulfills the requirement above, the first of them (and any otherfmapwith matching<acq>and<run>entities, so thatpepolarpairs and magnitude/phase pair have the same"IntededFor"runs) will get the"IntendedFor". In the future, we might want to change this (e.g., so that it uses the one closest in time), or we might want to give the user the option to choose one or the other.It addresses #474.