feat(source-apify-dataset): add option to specify dataset by its name

[vdusek]

What

• Introduce a new feature to specify an Apify dataset by its name.
• Closes: [source-apify-dataset] Reference dataset using its name rather than ID #43321

How

• Added two new properties: username and dataset_name. These are necessary to identify the dataset by its name.
• Implemented logic to handle dataset resolution based on the provided inputs. If dataset_id is provided, the dataset will be resolved using it. If username and dataset_name are provided, the dataset will be resolved using those instead. Fail otherwise.
  • This logic is a work in progress - any guidance or feedback would be appreciated (see ## Todo).
  • Tagging @flash1293 directly, as he helped me in the last updates of the connector (Source Apify Dataset: fix broken stream, manifest refactor #30428, Source Apify Dataset: add item_collection stream with dynamic schema #31333, Source Apify Dataset: improve connector documentation #31397), thanks.

Todo

• This PR is still under development, and I'm seeking assistance with the following challenge:

Logic for dataset resolution in DeclarativeStreams

• There should be two distinct branches for resolving the dataset path:

1. Using dataset_id:

path: "datasets/{{ config['dataset_id'] }}"

2. Using username and dataset_name:

path: "datasets/{{ config['username'] }}~{{ config['dataset_name'] }}"

If neither or both of these options are provided, the process should fail.

• Question: How can this conditional logic be defined in the manifest? Is there a way to enforce this directly?

User Impact

• This update introduces an alternative way to reference a dataset, but the existing method (using dataset_id) remains fully functional.

Can this PR be safely reverted and rolled back?

• YES 💚

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Aug 28, 2024 1:15pm

[marcosmarxm]

@vdusek, are you planning to continue working on this contribution? Today, you made a change in the spec, but it hasn't been applied to any part of the connector code.

[vdusek]

@marcosmarxm Hi, I'd love to. Have you read the "Logic for dataset resolution in DeclarativeStreams" section? I have a question there regarding the conditional evaluation in the manifest.

[omcknightkittl]

Hey, any update on this fix? Would be greatly appreciated :)

[vdusek]

Well, I haven't found anything about the conditional logic in the manifests in docs.

Another approach would be to remove the dataset_id parameter and replace it with dataset_name. This change would be breaking, so it would require releasing a new major version. However, this solution wouldn't work for users who want to use a dataset ID instead of a name.