Monte Carlo simulation of errors for ESA SCOPE.
Read INSTALL.md for detailed instructions on installation and testing.
Processors are coded in Python and require an environment described
in INSTALL.md. The kaleidoscope directory
includes the kaleidoscope Python package, which includes the Kaleidoscope
processors. The processors are invoked from the command line. Typing
kaleidoscope-scatter --help
kaleidoscope-collect --help
will print a detailed usage messages to the screen, like
usage: kaleidoscope-scatter [-h] --source-type
{esa-cci-oc,esa-scope-cs,esa-scope-pp,ghrsst,glorys}
--selector {0,1,2,...,100}
[--antithetic]
[--engine-reader {h5netcdf,netcdf4,zarr}]
[--engine-writer {h5netcdf,netcdf4,zarr}]
[--log-level {debug,info,warning,error,off}]
[--mode {multithreading,synchronous}]
[--workers {1,2,3,4,5,6,7,8}] [--progress]
[--stack-traces] [-v]
source_file target_file
This scientific processor produces a Monte Carlo ensemble from given
uncertainties.
positional arguments:
source_file the file path of the source dataset.
target_file the file path of the target dataset.
options:
-h, --help show this help message and exit
--source-type {esa-cci-oc,esa-scope-cs,esa-scope-pp,ghrsst,glorys}
the source type. (default: None)
--selector {0,1,2,...,100}
the Monte Carlo stream selector. An integral number
which must not be negative. (default: None)
--antithetic enable pairwise antithetic Monte Carlo simulation.
(default: False)
--engine-reader {h5netcdf,netcdf4,zarr}
specify the engine used to read the source product
file. (default: None)
--engine-writer {h5netcdf,netcdf4,zarr}
specify the engine used to write the target product
file. (default: None)
--log-level {debug,info,warning,error,off}
specify the log level. (default: None)
--mode {multithreading,synchronous}
specify the operating mode. In multithreading mode a
multithreading scheduler is used. In synchronous mode
a single-thread scheduler is used. (default: None)
--workers {1,2,3,4,5,6,7,8}
specify the number of workers used in multithreading
mode. If not set, the number of workers is determined
by the system. (default: None)
--progress enable progress bar display. (default: False)
--stack-traces enable Python stack traces. (default: False)
-v, --version show program's version number and exit
Copyright (c) Brockmann Consult GmbH, 2025. License: MIT
and
usage: kaleidoscope-collect [-h] --source-type
{esa-scope-dic,esa-scope-doc,esa-scope-pc,esa-scope-pic,esa-scope-poc,esa-scope-pp}
[--engine-reader {h5netcdf,netcdf4,zarr}]
[--engine-writer {h5netcdf,netcdf4,zarr}]
[--log-level {debug,info,warning,error,off}]
[--mode {multithreading,synchronous}]
[--workers {1,2,3,4,5,6,7,8}] [--progress]
[--stack-traces] [-v]
source_glob target_file
This scientific processor computes standard uncertainty from a given Monte
Carlo ensemble.
positional arguments:
source_glob the file path glob of the source datasets. The first
entry in the expanded list of file paths shall refer
to the nominal (i.e., not randomized) source dataset.
The remaining entries shall refer to randomized
variants of the nominal source. Only the '*' character
shall be used for globbing.
target_file the file path of the target dataset.
options:
-h, --help show this help message and exit
--source-type {esa-scope-dic,esa-scope-doc,esa-scope-pc,esa-scope-pic,esa-scope-poc,esa-scope-pp}
the source type. (default: None)
--engine-reader {h5netcdf,netcdf4,zarr}
specify the engine used to read the source product
file. (default: None)
--engine-writer {h5netcdf,netcdf4,zarr}
specify the engine used to write the target product
file. (default: None)
--log-level {debug,info,warning,error,off}
specify the log level. (default: None)
--mode {multithreading,synchronous}
specify the operating mode. In multithreading mode a
multithreading scheduler is used. In synchronous mode
a single-thread scheduler is used. (default: None)
--workers {1,2,3,4,5,6,7,8}
specify the number of workers used in multithreading
mode. If not set, the number of workers is determined
by the system. (default: None)
--progress enable progress bar display. (default: False)
--stack-traces enable Python stack traces. (default: False)
-v, --version show program's version number and exit
Copyright (c) Brockmann Consult GmbH, 2025. License: MIT
To invoke the processor from the terminal, for instance, type
kaleidoscope-scatter --source-type ghrsst --selector 17 in.nc out.nc
which normally will log information to the terminal, e.g.,
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] starting running processor
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: engine_reader = None
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: engine_writer = None
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: log_level = info
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: mode = multithreading
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: processor_name = kaleidoscope-scatter
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: processor_version = 2025.1.1
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: progress = False
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: selector = 17
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: source_file = in.nc
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: source_type = ghrsst
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: stack_traces = False
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: target_file = out.nc
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: tmpdir = .
2025-05-22T08:29:19.999000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] config: workers = 2
2025-05-22T08:29:20.108000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] starting creating processing graph
2025-05-22T08:29:20.110000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] starting graph for variable: analysed_sst
2025-05-22T08:29:23.228000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] finished graph for variable: analysed_sst
2025-05-22T08:29:23.228000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] finished creating processing graph
2025-05-22T08:29:23.228000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] starting writing target dataset: out.nc
2025-05-22T08:29:29.476000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] finished writing target dataset
2025-05-22T08:29:29.476000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] starting closing datasets
2025-05-22T08:29:29.477000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] finished closing datasets
2025-05-22T08:29:29.477000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] finished running processor
2025-05-22T08:29:29.477000Z <node> kaleidoscope-scatter 2025.2.0 [71491] [I] elapsed time (seconds): 9.478
and eventually produce a randomized output dataset. Normally, the processor
will terminate with an exit code of 0.
The processor terminates on the first occurrence of an error. The exit code
of the processor is 0 if the processor completed without errors, and nonzero
otherwise. Warning and error messages are sent to the standard error stream.
There are no recovery operations.
Kaleidoscope Scatter generates Monte Carlo variants of source data which exhibit simulated errors. At present, only uncorrelated random errors are simulated. Normalized errors will have a mean value of zero and a standard deviation of unity, with statistical deviations implied by constraints on the error probability distribution and the (limited) number of variants generated.
Examples above illustrate the mean, the standard deviation and the minimum-to-maximum range of normalized simulated errors for an ensemble of ten variants of global monthly sea surface temperature (January 2000). The standard deviation and range of normalized errors is reduced at the North Pole and in the Weddell Sea due to constraints imposed by the freezing point of seawater.
Kaleidoscope Collect generates standard uncertainty from Monte Carlo variants. Examples below illustrate the nominal primary production and its uncertainty. Standard uncertainty typically exhibits statistical fluctuation, which is an artifact of the Monte Carlo method. Therefore, a low-pass filtered standard uncertainty is produced in addition.
