`AddedFieldDefinition`: include option to use parent in `interpolation_context`

[marcosmarxm]

I want to include a field in my child stream using parent record information.
Example the API has orders and orderDetails endpoint.

Parent Record: orders

{
  "orderId": "12345",
  "customerName": "John Doe",
  "orderDate": "2024-12-06",
  "totalAmount": "150,75", 
  "status": "Shipped"
}

Child Record: orderDetails

{
  "orders": {
    "orderId": "12345",
    "orderDetails": [
      {
        "orderDetailId": "98765",
        "productName": "Wireless Mouse",
        "quantity": 2,
        "price": "25,50",
        "subtotal": "51,00"
      },
      {
        "orderDetailId": "98766",
        "productName": "Keyboard",
        "quantity": 1,
        "price": "99,99",
        "subtotal": "99,99"
      }
    ]
  }
}

Today is possible to include orderId because it is the key from the parent but if I want to include let's say status to orderDetail object it isn't possible.

[natikgadzhi]

There is a workaround for this today, but you need to make a custom column on the parent record that contains a json represenation of multiple columns you want to access. Then you just use that custom column as parent "pk", and can key into it from the child stream.

[nadavge]

To preserve knowledge, this is the most naive way to do it. This can also be done with specific fields, and then instead of {{ record }} you will need to do something like

{ "key1": "{{ record['value1'] }}", "key2": "{{ record['value2'] }}" }

Not sure how many edge cases this has (what happens for example if there are quotes in the value and so on. Handle with care.

Then on the child stream you define __airbyte_hack as the partitioning, and you can access a specific key with an extra bracket. Assuming __airbyte_hack is mapped to parent_data:

{{ stream_partition['parent_data']['key1'] }}