Possibility to include file_package_name(s) as part of metadata?

[johannesocean]

Feature Request

In order to be able to track the history of a given dataproduct (produced by satpy) filenames/name of packages would be usefull as part of metadata.

Ex.
I´m reading two scenes into satpy, merging them and then writes an nc-file.. When opening that nc-file I might want to know the names of the packages that produced that specific nc-file.

Possible solution:
Adding "file_name" (or "source" might be a better name) as an attribute to each variable? If multiple sources are loaded you include these as comma-seperated string like:
:source= "source_name_1, source_name_2"

[djhoese]

Good idea!

There is likely a CF standard way of defining this information like the source attribute described here: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/ch02s06.html
However, that doesn't seem to talk about input filenames.

[mraspaud]

Thanks for proposing this feature ! Do we want to aggregate all the files to a single attribute, or do we want a source attribute for each variable in the resulting netcdf file ?

Also, as I understand the CF document, maybe source isn't the right place for this ? but then I wouldn't know where to put it as history doesn't feel correct either.

[mraspaud]

ah:

comment

    Miscellaneous information about the data or methods used to produce it.

Should we use that ?

[johannesocean]

For my purposes, it would be neat to have that information aggregated in to a single attribute,, perhaps there are other source metadata that could be included apart from filenames? ( I'm looking at the global attributes of "instrument_data.nc" thats included in OLCI-Level 2 dataset )

[mraspaud]

Could you paste an example of what you would like ?

[djhoese]

This sounded familiar to me so I looked at some other work I've done with NetCDF files. The ARM (Atmospheric Radiation Measurement group) NetCDF standard uses an input_source global attribute to specify the first input file used to generate the current file (see https://www.arm.gov/publications/programdocs/doe-sc-arm-15-004.pdf). However, this is only one file and the ARM standard is now moving towards the CF standard more and more. Maybe we could ask the CF group what they would suggest?

[johannesocean]

Source as below with product_names specifing each package of the loaded data.. Not sure if this is the way to go.. the CF group that @djhoese mentioned might be the first step here?

ubyte chl_oc4me(y=4091, x=4865);
  :_FillValue = 255UB; // ubyte
  :calibration = "reflectance";
  :end_time = "2019-10-07 10:39:15.111166";
  :long_name = "log10 scaled (OC4ME) Algal pigment concentration";
  :modifiers = ; // double
  :platform_name = "Sentinel-3A";
  :resolution = 300L; // long
  :sensor = "olci";
  :start_time = "2019-10-07 10:36:15.146145";
  :units = "lg(re mg.m-3)";
  :coordinates = "longitude latitude";
  :add_offset = -2.0f; // float
  :scale_factor = 0.015748031f; // float
  :_ChunkSizes = 1364U, 1622U; // uint

short source(rows=4091, columns=4865);
  :contact = "ops@eumetsat.int";
  :institution = "MAR";
  :product_names = "S3A_OL_2_WFR____20191004T101049_20191004T101349_20191004T122429_0179_050_065_1800_MAR_O_NR_002.SEN3, S3A_OL_2_WFR____20191004T101349_20191004T101649_20191004T122334_0179_050_065_1980_MAR_O_NR_002.SEN3";
  :references = "S3IPF PDS 004.3 - i2r2 - Product Data Format Specification - OLCI Level 2 Marine, S3IPF PDS 002 - i1r7 - Product Data Format Specification - Product Structures, S3IPF DPM 008 - i1r3 - Detailed Processing Model - OLCI Level 2";
  :source = "IPF-OL-2 06.13";
  :start_time = "2019-10-04T10:10:48.570298Z";
  :stop_time = "2019-10-04T10:13:48.535319Z";
  :title = "OLCI Level 2 WATER Product, Instrument Data Set"