Metadata

Proposal for description of data formats/types

A format can be described uniquely with a triplet format-type-level.

format is the name of the format or set of conventions used to record the data in the file. Examples: HRPT, CHRPT, AHRPT, PDS
type is the type of the file where the data is recorded. Examples: binary, HDF4, HDF5, NetCDF

examples:

Format	Type	Level	Notes
HRPT	Binary	0	High resolution picture transmission format
CHRPT	Binary	0	Color HRPT
AHRPT	Binary	0	Advanced HRPT
PDS	Binary	0	Production data set
EOS	HDF4	1b or 2	eg Modis lvl1 files
RDR	HDF5	0	S-NPP (output of RT-STPS)
SDR	HDF5	1b
Cadu	Binary	0
EPS	Binary	1a, 1b, or 2	EPS native format
CF	NetCDF	anything	Pytroll output files
AAPP-HRPT	Binary	1b, 1c, 1d	AAPP output files
GeoTIFF	TIFF	3	Pytroll tiff output files
NinjoTIFF	TIFF	3	Pytroll Ninjo tiff output files
Raster	PNG, JPG, ...	3	Raster image files

So when publishing we should advertise the data as something like EOS/HDF4/1B.

ZMQ topics for Pytroll messages:

For errors and warnings: '''/alarm'''
For new products (files): '''/file-<format>-<level>[-<instrument>]''', e.g '''/file-HRPT-AAPP-1B-AVHRR'''
For status information: '''/status''', e.g '''/status/sat/polar/smb/level0''', including a json object with e.g a level0 status field.
For general information: '''/info'''

ZMQ topics for Pytroll messages: '''/<format>/<level>/'''

"platform_name" (string): the name of the satellite, eg: "NOAA-19", "Metop-A", "Suomi-NPP". Try to stick to post-commissioning names.

(I propose we drop the separate satellite and number identifiers, as they were artificially introduced to please mpop actually)

"platform_id" (string, optional): the international designator of the satellite, eq: "2009-005A", "2006-044A", "2011-061A"
"sensor" (string, or list of strings): instrument(s) that contribute to the dataset.
"orbit_number" (int, only for polar orbiters): the orbit number, tle style
"orbit_direction" (string, optional): "ascending" or "descending"
"start_time" (datetime): start of observation time
"end_time" (datetime): end of observation time
"nominal_time" (datetime): nominal time of observation.

(at least one of the times is needed to identify the dataset)

"uri" (string): location of the dataset
"data_processing_level" (string): level of the dataset, eg "1b", "0" (see here)
"format" & "type" (strings): as described above
"uid" (string): unique id for the file enclosing the dataset. Could be the md5sum of the file
"description" (string, optional): description of the dataset
"product_name" (string, optional): name of the product, eg "overview", "cloudtype"...
"area" (dict, optional): description of the area covered by the dataset, contains the following:
- "name" (string): free string describing the area
- "id" (string): id, more rigid (no white spaces for example)
- "proj4" (string): the proj4 string
- "area_extent" (list of 4 floats): the area extent of the area, in projection coordinates
- "shape" (list of 2 ints): the size in pixels of the area.

Following satellite names on oscar

Following sensor names on oscar

...