Using Apptainer/Singularity gives `FAILURE! hictk encountered the following error: filesystem error: temp_directory_path: No such file or directory`

[paulmenzel]

$ apptainer pull docker://ghcr.io/paulsengroup/hictk:1.0.0
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 7646c8da3324 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 4910288d28ee skipped: already exists  
Copying blob 6136142663cf skipped: already exists  
Copying config ef8f297598 done  
Writing manifest to image destination
Storing signatures
2024/07/11 09:32:38  info unpack layer: sha256:7646c8da332499ae416b15479ce832db32e39a501c662e24324f595509a0d3db
2024/07/11 09:32:39  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2024/07/11 09:32:39  info unpack layer: sha256:4910288d28eefbfd39b4a4447975da38c9d91a5e9158a5ac47bd0765caf6dada
2024/07/11 09:32:39  info unpack layer: sha256:6136142663cf665dff7c05fe33c3f69cec187b3c78a4bcdfc3850fb12528135c
2024/07/11 09:32:39  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
2024/07/11 09:32:39  info unpack layer: sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
INFO:    Creating SIF file...
$ apptainer run hictk_1.0.0.sif --version
INFO:    gocryptfs not found, will not be able to use gocryptfs
FAILURE! hictk encountered the following error: filesystem error: temp_directory_path: No such file or directory

[paulmenzel]

TMPDIR is set to /scratch/local in our environment, which does not seem to be available in the image. Setting TMPDIR to /tmp fixes the issue:

$ TMPDIR=/tmp apptainer run hictk_1.0.0.sif --help
INFO:    gocryptfs not found, will not be able to use gocryptfs
Blazing fast tools to work with .hic and .cool files.
Usage: /usr/local/bin/hictk [OPTIONS] SUBCOMMAND

Options:
  -h,--help                   Print this help message and exit
  -V,--version                Display program version information and exit

Subcommands:
  balance                     Balance Hi-C matrices using ICE, SCALE, or VC.
  convert                     Convert HiC matrices to a different format.
  dump                        Dump data from .hic and Cooler files to stdout.
  fix-mcool                   Fix corrupted .mcool files.
  load                        Build .cool and .hic files from interactions in various text formats.
  merge                       Merge multiple Cooler or .hic files into a single file.
  rename-chromosomes, rename-chroms
                              Rename chromosomes found in a Cooler file.
  validate                    Validate .hic and Cooler files.
  zoomify                     Convert single-resolution Cooler and .hic files to multi-resolution by coarsening.

If you think, it’s a non-issue, please close.

[robomics]

Thanks for reporting this issue and sharing a reasonable workaround!

I think hictk is doing the right thing by issuing an error when a temporary folder cannot be safely created, however I think the current error message is a bit cryptic for less experienced users.

I will keep this issue open as a reminder to output a better error message (something like failed to create a temporary folder: please make sure the environment variable TMPDIR is defined and pointing to an existing folder).

[paulmenzel]

Thank you for your quick response.

I will keep this issue open as a reminder to output a better error message (something like failed to create a temporary folder: please make sure the environment variable TMPDIR is defined and pointing to an existing folder).

I’d also add the value of the tried directory.

failed to create a temporary folder /scratch/local

[robomics]

The fix for this issue should be part of the next release.

On my machines these are the new errors generated by hictk:

TMPDIR points to a non-existing folder:

TMPDIR="foobar" hictk --help
FAILURE! hictk encountered the following error: filesystem error: unable to safely determine the path where to store temporary files: temporary folder is set to "foobar" but folder does not exists: No such file or directory

TMPDIR is empty or undefined

TMPDIR="" hictk --help
FAILURE! hictk encountered the following error: filesystem error: unable to safely determine the path where to store temporary files: please make sure the environment variable TMPDIR is defined and pointing to an existing folder: No such file or directory