compressed file access through rsync needs to be more robust

[blanton144]

Right now the RsyncAccess.add() method appends '*', which catches compressed files but can include other undesired files.

For example the following example downloads the spAll file AND a bunch of the backup files:

import sdss_access
sdss_rsync = sdss_access.RsyncAccess(release='sdsswork', verbose=True)
sdss_rsync.remote()
sdss_rsync.add('spAll', run2d='v6_2_0')
sdss_rsync.set_stream()
sdss_rsync.commit()

[Sean-Morrison]

The example case here is mitigated for the time being, the backup files were renamed/moved so they do not match the wildcard. It is still a bug that would be good to solve in long run

[johndonor3]

I think I found a related issue? If you create a local path access.path.Path.full then check access.path.Path.exists(localname) and the file is not found, so you use remote(), etc, to fetch the file, there is a mismatch between the initial file name and the file that is fetched. Specifically: the initial access.path.Path.full is a .fits file path, but .fits.gz is downloaded.

The issue crops up a few times if you run this notebook https://github.com/sdss/dr19_tutorials/blob/main/allspec.ipynb without any of those summary files existing locally.

[joelbrownstein]

tree doesn't encode the gz in the file path; but sdss_access is "smart" about that and adds the gz if needed.
So this is a feature of sdss_access, not a bug; but may require we add a smart utility function to solve the issue you're seeing with those tutorials

[johndonor3]

I guess if we did another path.full after running the download commands it would pick up the .gz? Is that right?

I'm realizing that the way it's written, we don't give sdss-access a chance to check if it needs .gz; we just hand the original path name straight to fits.open() so maybe this is user error at some level?

[havok2063]

sdss_access needs some logic to handle this since we're inconsistent with files on disk and the path definitions being compressed or not. It's also a bi-directional issue. path.full should always return the filepath as it exists on disk, regardless of the original template or file compression.