Skip to content

query: syntax to include metadata as part of the program. #6

@prataprc

Description

@prataprc

Query, being inspired from JQ, allows the program to be
applied on one or more input document, where document can be
any json encoded value (or encoded in json equivalent format).
While this is useful to process JSON value in sed and
awk like fashion, it has some serious short-comings
when used in SQL fashion.

  • Key, big data systems are predominanty based on {key,value}
    document storage as its primary storage. To that extent we need
    to include key as part of the query-program.

  • Metadata, at big data scale, documents are predominantly
    distributed. This along with features like, append only values,
    create-modify-read timestamps, uuid, document versioning etc..
    adds one or more metadata fields to the actual document-value.

The solution we are proposing is:

  • To keep metadata separate from actual document value.
  • To include both document and metadata as inputs to query program.
  • Right now query programs can access the document value using
    . (dot) syntax, here after we can allow query programs to use # (hash)
    syntax to access the documents metadata.
  • Key can be made available via metadata, like #_key.
  • While document (.) can be any JSON value, metadata is always
    a set of {key,value} pairs, where Key is of string-type and value
    can be any valid JSON value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions