FIX: Use copy function that does not preserve atime when creating fsaverage directories

[effigies]

Closes #702.

[codecov-commenter]

Codecov Report

Merging #703 (26873d0) into maint/1.3.x (35b6435) will not change coverage.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##           maint/1.3.x     #703   +/-   ##
============================================
  Coverage        48.06%   48.06%           
============================================
  Files               43       43           
  Lines             5297     5297           
  Branches           776      776           
============================================
  Hits              2546     2546           
  Misses            2663     2663           
  Partials            88       88           

Flag	Coverage Δ
reportlettests	∅ <ø> (∅)
unittests	48.06% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
niworkflows/interfaces/bids.py	96.15% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 35b6435...26873d0. Read the comment docs.

[effigies]

@mgxd Would appreciate a review to make sure I'm not crazy. I intend to do a release cascade so we can be prepared for the next round of fMRIPrep releases.

[mgxd]

It doesn't seem to matter which copy function is used (at least from local testing on my Macbook), the directory mtime stays the same. In fact, the only timestamp that changes is ctime, and that happens regardless of the copy function.

If sweeps take that into account, we're probably fine - if not, we could just hack the mtime with something like

 os.utime(dest, (time.time(), time.time()))

[effigies]

I can reproduce the issue on Linux. Want to try this script?

Script:

#!/bin/bash

set -eux

mkdir testdir
touch -d "Jan1" testdir/testfile
echo "atime"
ls -luR testdir
echo "mtime"
ls -lR testdir

python <<EOF
import shutil

shutil.copytree("testdir", "testdir.copy2", copy_function=shutil.copy2)
EOF

# Reset atime from copytree
touch -d "Jan1" testdir/testfile

python <<EOF
import shutil

shutil.copytree("testdir", "testdir.copy", copy_function=shutil.copy)
EOF

echo "atime"
ls -lu testdir.copy/testfile
ls -lu testdir.copy2/testfile
echo "mtime"
ls -l testdir.copy/testfile
ls -l testdir.copy2/testfile

Result:

+ mkdir testdir
+ touch -d Jan1 testdir/testfile
+ echo atime
atime
+ ls -luR testdir
testdir:
total 0
-rw-rw-r-- 1 chris chris 0 Jan  1 00:00 testfile
+ echo mtime
mtime
+ ls -lR testdir
testdir:
total 0
-rw-rw-r-- 1 chris chris 0 Jan  1 00:00 testfile
+ python
+ touch -d Jan1 testdir/testfile
+ python
+ echo atime
atime
+ ls -lu testdir.copy/testfile
-rw-rw-r-- 1 chris chris 0 Mar 30 21:16 testdir.copy/testfile
+ ls -lu testdir.copy2/testfile
-rw-rw-r-- 1 chris chris 0 Jan  1 00:00 testdir.copy2/testfile
+ echo mtime
mtime
+ ls -l testdir.copy/testfile
-rw-rw-r-- 1 chris chris 0 Mar 30 21:16 testdir.copy/testfile
+ ls -l testdir.copy2/testfile
-rw-rw-r-- 1 chris chris 0 Jan  1 00:00 testdir.copy2/testfile

[mgxd]

I think we're looking at this in two different ways.

Your tests are comparing the files within testdir, which absolutely show how using shutil.copy over shutil.copy2 modifies the mtime.

I was looking at the top level directory's mtime after using shutil.copy / shutil.copy2. My test was copying an existing fsaverage* directory, and then checking the copied directories' mtime. I'm not sure if the sweeps also care about directories, otherwise this is fine to merge.

[effigies]

The relevant bit is actually the atime more than the mtime. What we think is happening is that the sweeps are seeing files that haven't been accessed in years/months and suddenly they're gone. Being in a more recently accessed or modified directory does not seem to protect them.

Hope this is more clear. Can chat tomorrow if we're still not quite connecting.

[mgxd]

Ah, then this LGTM