Skip to content

Home

Jump to bottom
Sean Solberg edited this page Dec 29, 2023 · 5 revisions

Welcome to the DynamicDataObjects wiki!

What is Data Objects

Dynamic Data Objects is a library that makes it easy to model complex structures of data in memory. Then, you can easily serialize to/from it using a variety of data serialization techniques including: CBOR, JSON, MessagePack, BSON, UBJSON, Smile, Amazon's ION (binary style only), csv, ics calendars, and others.

There are two main co-dependent aspects to Dynamic Data Objects that are at play here.

In-Memory Object Model: DataObjects is a library of code that provides a few classes that can be used to model a dynamic hierarchical structure of data in memory. In concept, it's similar to a "variant" or "oleVariant" data type that's available in some programming languages. A dataObject can hold a value in memory and that value can be one of many possible data types such as a signed integer, a double floating point number, a string, etc. It also has a few different types of collections like arrays or maps (key-value collections). DataObjects supports two key-value collections types: Sparse Arrays (a key-value map where the key is an integer), and a Frame (a key-value map where the key is a string. These are sometimes called dictionaries, maps, etc.)
Serialization: The primary purpose here is to take a DataObject that may hold a complex hierarchy of data in memory and serialize it into a stream of bytes that can be sent over a network, saved to a file, etc. It could be used for server-to-server communications, client-to-server communication, data or preference persistence, etc. Ultimately, however, the DataObject can be serialized to a stream and another DataObject elsewhere can read that stream and built an exact in-memory clone of the original. In most cases, this is done without any data loss or data translation. However, some of the serializations supported, such as JSON, do a very poor job of modeling data, so when using certain serializations, there must be some translations taking place. For example, a GUID data value will be serialized in JSON as a string because JSON doesn't have native support for the GUID data type.

Note that the performance of this library has been heavily considered on most of the serializers, particularly JSON, CBOR, BSON and DataObj as these are the four that I use the most. JSON in particular has been heavily performance tested with other JSON parsers and this library beats the others under most conditions (not 100% though). In general, you will find it to perform very well.

Note about the DataObj and DDO serializations. The DDO serializer mostly supports an older proprietary file format that I used in the past, except that it doesn't support the geometry data types that are possible in a DDO file. No one should ever use this serializer as its format is old and there are no benefits to using it. I just have it to be able to read old config files. The DataObj serializer is the one that I use as a standard format. It is the one that is most closely paired with the TDataObj object model and it is considered the default. It supports all the data types natively, it is fast, and it produces the most compact byte stream.

Implementation

This is a Delphi library. I am making sure it works correctly for Win32 and Win64 right now. Eventually I'll make sure it works for Android, IOS and Linux too but not until I think the first version is solid enough to start putting time into testing with these other targets.

The only thing you need to do use the core of data objects is to include "dataObjects2" in a uses clause.

For each serialization you want to use, just include the associated unit for it in a uses clause.

That's it. No dlls needed. No other dependencies.

I do have a few additional utilities included in the source such as a Save-As dialog that understands the registry of serializers that are included in a project.

This code might work in other pascal compilers such as Lazarus/free pascal, but I don't know for sure as that hasn't been tested.

Use Cases

The most common use cases that I use this library for are:

Syncing structured data models between servers. Pulling data from MongoDB using BSON. Publishing HTTP services to browser javascript clients using JSON. Persisting complex preferences for an application instead of using an *.ini file or the registry. Syncing structured data models and calling remote procedure calls between a client and a server. Saving logs that have a bit more complex data than just text. That way I can mine the logs later if I want to extract good data and do analytics on something.

Pre-compiled Tools

There are 3 compiled binary executables within this repository that are built with the DataObjects library as follows:

  1. DataObjectInstaller32.exe - This is a simple nsis installer that will install the three tools listed below. This is the easiest way to get everything setup correctly as it will register the supported file formats with windows to easily edit these files with the editor and it will register the explorer plugin.

  2. DataObjectEditor.exe - This is a simple windows executable that will open any of the files supported by the dataObjects serializers. It is very fast and can handle very large files in the tree-orientated UI. This tool supports editing and saving, etc. but note that it is still a work in progress and it is lacking some of the more advanced editing features that are planned.

  3. DataObjPreviewer.dll - This is a windows explorer preview plugin that will show the contents of any of the supported formats in the windows explorer preview side-panel. This tool is super helpful, especially if you are frequently browsing json files. This must be registered with "regsvr32 DataObjPreviewer.dll" but if you use the installer, then that is done automatically.

  4. DataObj.exe - This is a simple command line tool that lets you convert from any of the supported formats to any other format from within the command shell. For example: ".\DataObj.exe -i *.cbor -f json -od c:\temp" will take all cbor files in the current directory and convert them to corresponding json files and place them into the c:\temp directory. Just type .\DataObj.exe and it will produce help information on how to use it.

Clone this wiki locally