DocumentDb simple representations of BSON types

[dlvenable]

Problem/Background

The upcoming documentdb source is putting BSON types directly into the Event model. These types are not supported by downstream processors or sinks. So at the best, they will get the toString() representations.

Solution

When constructing the Data Prepper event in the documentdb source, create simple representations of BSON types.

ObjectId

For the BSON ObjectId, make this the string representation.

Input:

{
  "name" : "Star Wars",
  "directorId" : ObjectId("fdd898945")
}

Output from documentdb source:

{
  "name" : "Star Wars",
  "directorId" : "fdd898945"
}

BinData

For the BSON BinData type, make the simple representation a base64 encoded string for that binary data. The subtype will be lost.

Input:

{
  "filepath" : "/usr/share/doc1",
  "myBinary" : BinData(binary="[0x88]", subtype="MD5")   # Not actual format; just a conceptual representation
}

Output:

{
  "filepath" : "/usr/share/doc1",
  "myBinary" : "X7ah=="
}

Timestamp

For the BSON Timestamp type, create an integer with the epoch seconds. The ordinal will be lost.

Input:

{
  "name" : "Star Wars",
  "lastUpdatedAt" : Timestamp(time=1713536835 ordinal=12)   # Not actual format
}

Output:

{
  "_id" : "abcdef12345",
  "name" : "Star Wars",
  "lastUpdatedAt" : 1713536835    # The time part. The ordinal is lost
}

Regular Expressions

The BSON RegEx type has both a pattern and options part. I'm unsure if the patterns would be sufficient on its own. So perhaps this one must have nested fields.

Input:

{
  "use-case" : "email_address",
  "myRegex" : RegEx(pattern="[A-Za-z0-9+_.-]+@([\w-]+\.)+[\w-]{2,4}", options="i")    # Not actual format
}

Output:

{
  "use-case" : "email_address",
  "myRegex" : {
    "pattern" : "[A-Za-z0-9+_.-]+@([\w-]+\.)+[\w-]{2,4}",
    "options" : "i"
  }
}

[dinujoh]

#4499