-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Frontend part of JSON backend implementation
Implement JSON backend to dummy state Misses the actual implementation of AbstractIOHandlerImpl Declare IOHandlerImpl for JSON and integrate it with other places Misses the implementation. Undebugged minimum implementation for JSON writing First basically runnable version of JSON writing To address: No reading or deleting yet. Datatypes are currently ignored and the data is assumed to be int64_t. Attribute values are ignored and replaced with a dummy value. If a subgroup name can be parsed as a nonnegative string, the JSON API will create a JSON array rather than a JSON object (associative array) as intended. Correctly handle groups that can be parsed as int See last commit's description. Fix index calculation with offsets in WriteData Fix some mistakes in JSON writing Correctly handle overwriting files: -> overwritten files should not be possible to access any longer and be clearly distinguished from the newly-created file Make some verifications execute independent of compiler options. Full implementation of JSON writing Respects all datatypes now. Format code according to Clion Stylesheet https://github.com/ComputationalRadiationPhysics/contributing/blob/master/IDESettings/CLion/CRP_CLion2016_1.xml Add generic branching over an openPMD datatype First runnable version of JSON Reading Cleanup and implementation of dataset extension Undebugged version of JSON deletion Properly (de)serialize datatypes Instead of casting the Datatype enum to and from int (which is likely to break when altering the enum), serialize to and from String values. Fix a number of mistakes in JSON reading and writing Cleanup Add JSON tests and fix bugs found thusly Add further tests and fix a further bug The JSON library does not understand long double values (i.e. 128bit floats), represent them as a char array. Handle floating point special values JSON represents +/-Infinity and NaN values as null. The JSON library will correctly serialize those values *to* JSON, implement (semi)-correct handly for deserialization. As it is unclear which exact value a null represents, deserialize it to NaN. Take notice that large floating point values (128 bit) might be serialized to null as well. Use std::is_floating_point to distinguish them from other types Additionally write the byte width of the underlying type Not yet used in reading Mark the writable written after successfully extending a dataset Remove support for absolute paths from openPath Fix some rough edges from rebasing Add documentation for the JSON backend Integrate the JSON backend with the build system Make platform bytewidth information global per JSON file Was previously annotated for every single Dataset and Attribute. Fixes -> shadowed variables -> return file streams by shared pointer so that Clang won't be mad about missing copy constructors -> handle inconsistencies between GCC, Clang and MSVC in switchType function Add licensing information and cleanup imports
- Loading branch information
1 parent
9b4c34f
commit 36ccc25
Showing
26 changed files
with
4,924 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| .. _backends-json: | ||
|
|
||
| JSON Backend | ||
| ============ | ||
|
|
||
| openPMD supports writing to and reading from JSON files. | ||
| For this, the installed copy of openPMD must have been built with support for the JSON backend. | ||
| To build openPMD with support for JSON, use the CMake option ``-DopenPMD_USE_JSON=ON``. | ||
| For further information, check out the :ref:`installation guide <install>`, | ||
| :ref:`build dependencies <development-dependencies>` and the :ref:`build options <development-buildoptions>`. | ||
|
|
||
|
|
||
| JSON File Format | ||
| ---------------- | ||
| A JSON file uses the file ending ``.json``. The JSON backend is chosen by creating | ||
| a ``Series`` object with a filename that has this file ending. | ||
|
|
||
| The top-level JSON object is a group representing the openPMD root group ``"/"``. | ||
| Any **openPMD group** is represented in JSON as a JSON object with three keys: | ||
|
|
||
| * ``attributes``: Attributes associated with the group. | ||
| * ``subgroups``: A JSON array of groups that appear below the current group. | ||
| * ``datasets``: A JSON array of datasets contained in the group. | ||
|
|
||
| Any of these keys may point to ``null`` or not be present, | ||
| thus representing an empty array / object. | ||
|
|
||
| Additionally to the three mentioned keys, the top-level group stores information about | ||
| the byte widths specific to the writing platform behind the key ``platform_byte_widths``. | ||
| Will be overwritten every time that a JSON value is stored to disk, hence this information | ||
| is only available about the last platform writing the JSON value. | ||
|
|
||
| Any **openPMD dataset** is a JSON object with four keys: | ||
|
|
||
| * ``attributes``: Attributes associated with the dataset. May be ``null`` or not present if no attributes are associated with the dataset. | ||
| * ``datatype``: A string describing the type of the stored data. | ||
| * ``extent``: A JSON array describing the extent of the dataset in every dimension. | ||
| * ``data`` A nested array storing the actual data in row-major manner. | ||
| The data needs to be consistent with the fields ``datatype`` and ``extent``. | ||
|
|
||
| **Attributes** are stored as a JSON object with a key for each attribute. | ||
| Every such attribute is itself a JSON object with two keys: | ||
|
|
||
| * ``datatype``: A string describing the type of the value. | ||
| * ``value``: The actual value of type ``datatype``. | ||
|
|
||
| Restrictions | ||
| ------------ | ||
| For creation of JSON serializations (i.e. writing), the restrictions of the JSON backend are | ||
| equivalent to those of the `JSON library by Niels Lohmann <https://github.com/nlohmann/json>`_ | ||
| used by the openPMD backend. | ||
|
|
||
| Numerical values, integral as well as floating point, are supported up to a length of | ||
| 64 bits. | ||
| Since JSON does not support special floating point values (i.e. NaN, Infinity, -Infinity), | ||
| those values are rendered as ``null``. | ||
|
|
||
| Instructing openPMD to write values of a datatype that is too wide for the JSON | ||
| backend does *not* result in an error: | ||
| * If casting the value to the widest supported datatype of the same category (integer or floating point) | ||
| is possible without data loss, the cast is performed and the value is written. | ||
| As an example, on a platform with ``sizeof(double) == 8``, writing the value | ||
| ``static_cast<long double>(std::numeric_limits<double>::max())`` will work as expected | ||
| since it can be cast back to ``double``. | ||
| * Otherwise, a ``null`` value is written. | ||
|
|
||
| Upon reading ``null`` when expecting a floating point number, a NaN value will be | ||
| returned. Take notice that a NaN value returned from the deserialization process | ||
| may have originally been +/-Infinity or beyond the supported value range. | ||
|
|
||
| Upon reading ``null`` when expecting any other datatype, the JSON backend will | ||
| propagate the exception thrown by Niels Lohmann's library. | ||
|
|
||
| A parallel (i.e. MPI) implementation is *not* available. | ||
|
|
||
| Example | ||
| ------- | ||
| The example code in the :ref:`usage section <usage-serial>` will produce the following JSON serialization | ||
| when picking the JSON backend: | ||
|
|
||
| .. literalinclude:: json_example.json | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| { | ||
| "attributes": { | ||
| "basePath": { | ||
| "datatype": "STRING", | ||
| "value": "/data/%T/" | ||
| }, | ||
| "iterationEncoding": { | ||
| "datatype": "STRING", | ||
| "value": "groupBased" | ||
| }, | ||
| "iterationFormat": { | ||
| "datatype": "STRING", | ||
| "value": "/data/%T/" | ||
| }, | ||
| "meshesPath": { | ||
| "datatype": "STRING", | ||
| "value": "meshes/" | ||
| }, | ||
| "openPMD": { | ||
| "datatype": "STRING", | ||
| "value": "1.1.0" | ||
| }, | ||
| "openPMDextension": { | ||
| "datatype": "UINT", | ||
| "value": 0 | ||
| } | ||
| }, | ||
| "platform_byte_widths": { | ||
| "BOOL": 1, | ||
| "CHAR": 1, | ||
| "DOUBLE": 8, | ||
| "FLOAT": 4, | ||
| "INT": 4, | ||
| "LONG": 8, | ||
| "LONGLONG": 8, | ||
| "LONG_DOUBLE": 16, | ||
| "SHORT": 2, | ||
| "UCHAR": 1, | ||
| "UINT": 4, | ||
| "ULONG": 8, | ||
| "ULONGLONG": 8, | ||
| "USHORT": 2 | ||
| }, | ||
| "subgroups": { | ||
| "data": { | ||
| "subgroups": { | ||
| "1": { | ||
| "attributes": { | ||
| "dt": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| }, | ||
| "time": { | ||
| "datatype": "DOUBLE", | ||
| "value": 0 | ||
| }, | ||
| "timeUnitSI": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| } | ||
| }, | ||
| "subgroups": { | ||
| "meshes": { | ||
| "datasets": { | ||
| "rho": { | ||
| "attributes": { | ||
| "axisLabels": { | ||
| "datatype": "VEC_STRING", | ||
| "value": [ | ||
| "x" | ||
| ] | ||
| }, | ||
| "dataOrder": { | ||
| "datatype": "STRING", | ||
| "value": "C" | ||
| }, | ||
| "geometry": { | ||
| "datatype": "STRING", | ||
| "value": "cartesian" | ||
| }, | ||
| "gridGlobalOffset": { | ||
| "datatype": "VEC_DOUBLE", | ||
| "value": [ | ||
| 0 | ||
| ] | ||
| }, | ||
| "gridSpacing": { | ||
| "datatype": "VEC_DOUBLE", | ||
| "value": [ | ||
| 1 | ||
| ] | ||
| }, | ||
| "gridUnitSI": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| }, | ||
| "position": { | ||
| "datatype": "VEC_DOUBLE", | ||
| "value": [ | ||
| 0 | ||
| ] | ||
| }, | ||
| "timeOffset": { | ||
| "datatype": "FLOAT", | ||
| "value": 0 | ||
| }, | ||
| "unitDimension": { | ||
| "datatype": "ARR_DBL_7", | ||
| "value": [ | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0 | ||
| ] | ||
| }, | ||
| "unitSI": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| } | ||
| }, | ||
| "data": [ | ||
| [ | ||
| 0, | ||
| 1, | ||
| 2 | ||
| ], | ||
| [ | ||
| 3, | ||
| 4, | ||
| 5 | ||
| ], | ||
| [ | ||
| 6, | ||
| 7, | ||
| 8 | ||
| ] | ||
| ], | ||
| "datatype": "DOUBLE", | ||
| "extent": [ | ||
| 3, | ||
| 3 | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| .. usage-firststeps: | ||
| .. _usage-firststeps: | ||
|
|
||
| First Steps | ||
| =========== | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| .. usage-parallel: | ||
| .. _usage-parallel: | ||
|
|
||
| Parallel API | ||
| ============ | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| .. usage-serial: | ||
| .. _usage-serial: | ||
|
|
||
| Serial API | ||
| ========== | ||
|
|
||
Oops, something went wrong.