Skip to content

Support compression in read_cmdstan_csv #1027

Open
@jsks

Description

@jsks

Cmdstan output files can get pretty large. data.table::fread used in read_cmdstan_csv and read_csv_metadata supports gzip and bzip2 compressed files if the R.utils package is installed.

On the cmdstanr side this would require at least modifying the suffix assertions (assert_file_exists) in both functions. And, switching out grep in favor of zgrep to handle the commented headers with the caveat (at least on linux) that the argument order be switched so that --color=never comes before the regex pattern.

So in read_cmdstan_csv:

fread_cmd <- paste0("zgrep --color=never -v '^#' '", path.expand(output_file), "'")

read_metadata_csv:

fread_cmd <- paste0("zgrep --color=never '^[#a-zA-Z]' '", path.expand(csv_file), "'")

and similarly for the WSL specific code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions