Support multiple datetime format for cursor field

[maxi297]

What area the feature impact?

Connectors

Revelant Information

Business Central API seems to return different datetime format for the cursor field: %Y-%m-%dT%H:%M:%S.%fZ and %Y-%m-%dT%H:%M:%SZ. This means that configuring an incremental sync is not possible for this API.

Proposed solution

Allow for multiple datetime formats. If multiple datetime_format are provided, start_datetime and end_datetime needs to define a datetime_format

  incremental_sync:
    type: "DatetimeBasedCursor"
    start_datetime:
      datetime: {{ config['start_date'] }}
      datetime_format: "%Y-%m-%d %H:%M:%S"
    end_datetime:
      datetime: {{ config['end_date'] }}
      datetime_format: "%Y-%m-%d %H:%M:%S"
    datetime_format: "%Y-%m-%dT%H:%M:%SZ"
    cursor_datetime_format: 
      - "%Y-%m-%dT%H:%M:%SZ"
      - "%Y-%m-%dT%H:%M:%S.%fZ"
    cursor_field: "publishedAt"

Dependency

If we allow datetime_format to be a list, it doesn't align with #27153. Possible solutions:

• do this issue before so that cursor_datetime_format can be a list and not datetime_format
• have outgoing_datetime_format instead of cursor_datetime_format and keep datetime_format for the cursor

[maxi297]

Grooming notes:

• cursor_datetime_format should be in order so that if the connector dev knows that one is more frequent than the other, he can defined it first and this will be more performant
• to avoid breaking changes, if cursor_datetime_format is not defined, use datetime_format
• cursor_datetime_format is a List[str]