Skip to content

CDK: Add note about nested cursor field #10897

Open
@alafanechere

Description

Nested cursor field does not currently work for normalization

Hi!

We are building a connector that syncs data from an internal api. We want it to load data incrementally, but we are struggling with defining the cursor field. The cursor field is in a nested structure:

    "dateto": "2036-01-01T23:59:59",
    "datefrom": "2022-01-03T04:30:00",
    "modidate": "2022-01-03T10:27:26",
    "dateupdate": "2022-01-03T04:30:00"
  }```
As per documentation (https://docs.airbyte.com/understanding-airbyte/catalog#cursor), we are setting `default_cursor_field` to `["dates", "modidate"]`.  We are getting an error in normalization though:

```2022-01-12 15:12:12 [42mnormalization[0m > ValueError: Unsupported nested cursor field dates.modidate for stream articles_stream```
We tried numerous things to debug, but with no luck. Do you have an idea what we are doing wrong?

Created from Slack: https://airbytehq.slack.com/archives/C01VDDEGL7M/p1642080469178000?thread_ts=1642080469.178000&cid=C01VDDEGL7M

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions