feat: IgnorableAPIError to compliment RetriableAPIError and FatalAPIError

[sebastianswms]

Feature scope

Taps (catalog, state, stream maps, etc.)

Description

Currently, the validate_response() method in rest.py can throw two types of errors: RetriableAPIError and FatalAPIError. I propose a new type of error in addition to these two: IgnorableAPIError. This error would indicate that:

• The response being processed is invalid.
• The endpoint should not be retried.
• The tap should not fail.
• The record being processed should be skipped—no data is returned.
• The tap should continue executing.

If an override of validate_response() was to throw a IgnorableAPIError, the instance would merely be logged and no further action would be taken.

AutoIDM has had multiple use cases for this functionality before when writing taps for applications such as Google Ads or GitLab. In the past, we've done custom implementations, but it seems like something with wide enough applicability that it should be baked into the SDK.

[edgarrmondragon]

Related:

• Let taps continue to next partitions and/or streams upon certain errors #280
• Draft: Allow skipping a partition by raising a specific exception - [opened] #635

[edgarrmondragon]

Thanks for logging @sebastianswms! A few folks have requested this (see #280).

Would be willing to contribute it to the SDK?

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant