Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buliding a source and facing issue at the stop condition #28659

Open
saikumar1182 opened this issue Jul 25, 2023 · 0 comments
Open

Buliding a source and facing issue at the stop condition #28659

saikumar1182 opened this issue Jul 25, 2023 · 0 comments

Comments

@saikumar1182
Copy link

Topic

At Stop condition in pagination

Revelant information

Hello,
I was trying to implement the api stream using the cursor pagination and giving the cursor value based on the cursor value start data end end date.
My yaml pagination part is

paginator:
        type: DefaultPaginator
        page_token_option:
          type: RequestOption
          inject_into: request_parameter
          field_name: searchAfter
        pagination_strategy:
          type: CursorPagination
          cursor_value: '{{ last_records[-1][''siren''] }}'
          stop_condition: '{{ ''record.siren'' not in response.body.siren | length == 0 }}'
    transformations:
      - type: AddFields
        fields:
          - path:
              - siren
            value: '{{ record[''company''][''formality''][''siren''] }}'
    incremental_sync:
      type: DatetimeBasedCursor
      cursor_field: company.updatedAt
      datetime_format: '%Y-%m-%d'
      start_time_option:
        type: RequestOption
        field_name: from
        inject_into: request_parameter
      end_time_option:
        type: RequestOption
        field_name: to
        inject_into: request_parameter
      start_datetime:
        type: MinMaxDatetime
        datetime: '{{ config[''start_date''] }}'
        datetime_format: '%Y-%m-%dT%H:%M:%SZ'
      end_datetime:
        type: MinMaxDatetime
        datetime: '{{ config[''end_date''] }}'
        datetime_format: '%Y-%m-%dT%H:%M:%SZ'

I would like to have a stop condition when it reaches the searchAfter cursor to the end of the siren value where no date is available.
Can someone help me defining the stop condition please.
Also i am receiving and infinite loop like

2023-07-25 10:12:51 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://registre-national-entreprises.inpi.fr/api/companies/diff?searchAfter=977955038&from=2023-07-01&to=2023-07-24, Response Code: 500, Response Text: {"code":"500","message":"Key not found: 4","type":"serverError"})
airbyte-worker          | 2023-07-25 10:12:51 source > Retrying. Sleeping for 10 seconds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants