Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fix for parse dataest definition #3400

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Dec 7, 2023

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

#3272 introduce an unexpected bug. #3369 is the original PR that try to address this.

The actual problem is that we accidentally drop the support of type: PythonClass (we never officially support it). In our docs we always expect it to support the following 3 cases. The decision is to not make any behavior change and we will add documentations and tests to support this.

dataset: Underlying dataset definition. This is used to instantiate
the dataset for each file located inside the path.
Accepted formats are:
a) object of a class that inherits from AbstractDataset
b) a string representing a fully qualified class name to such class
c) a dictionary with type key pointing to a string from b),

Development notes

  • add tests to support the change explicitly
  • add documentations.

How to test

  1. clone this branch and install it.
  2. run kedro-datasets tests.

If you running this on GitPod, you need some extra instruction to run datasets tests.

sudo apt-get install -y --no-install-recommends libatk-bridge2.0-0 libcups2 ca-certificates fonts-liberation libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils
sudo apt-get update && sudo apt-get install -y --no-install-recommends libgl1

I opened a separate PR to fix the gitpod docker to automate this.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

  • Read the contributing guidelines
  • Signed off each commit with a Developer Certificate of Origin (DCO)
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes
  • Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam noklam changed the base branch from main to develop December 7, 2023 17:05
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam noklam marked this pull request as ready for review December 7, 2023 17:14
@noklam noklam requested a review from merelcht as a code owner December 7, 2023 17:14
@noklam noklam marked this pull request as draft December 7, 2023 17:20
@noklam noklam marked this pull request as ready for review December 7, 2023 17:22
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
Copy link
Member

@merelcht merelcht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Nok! ⭐

Copy link
Contributor

@AhdraMeraliQB AhdraMeraliQB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested manually and did not run into any UnboundLocalError, thanks for the fix!

@noklam noklam merged commit 0b45b7f into develop Dec 11, 2023
30 checks passed
@noklam noklam deleted the noklam/parse-dataset-definition-support-both-str-and-python-object branch December 11, 2023 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants