Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KED-2315] Versioning datasets fails with NotADirectoryError #625

Closed
clstaudt opened this issue Nov 28, 2020 · 4 comments
Closed

[KED-2315] Versioning datasets fails with NotADirectoryError #625

clstaudt opened this issue Nov 28, 2020 · 4 comments
Labels
Issue: Bug Report 🐞 Bug that needs to be fixed

Comments

@clstaudt
Copy link

Description

Adding the versioned: true attribute to a dataset results in pipeline failure.

Context

The dataset versioning feature is not usable.

Steps to Reproduce

add versioned: true to a dataset in the catalog

Expected Result

A directory named like the dataset should have been created, containing versions of the dataset.

Actual Result

fails with NotADirectoryError

2020-11-28 18:48:01,270 - kedro.io.data_catalog - INFO - Saving data to `report_mass` (ExcelDataSet)...
2020-11-28 18:48:30,671 - kedro.runner.sequential_runner - WARNING - There are 1 nodes that have not run.
You can resume the pipeline run by adding the following argument to your previous command:
  --from-nodes "calculate_mass_sold"
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/kedro/io/core.py", line 240, in save
    self._save(data)
  File "/opt/anaconda3/lib/python3.7/site-packages/kedro/extras/datasets/pandas/excel_dataset.py", line 177, in _save
    with self._fs.open(save_path, **self._fs_open_args_save) as fs_file:
  File "/opt/anaconda3/lib/python3.7/site-packages/fsspec/spec.py", line 771, in open
    **kwargs
  File "/opt/anaconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 116, in _open
    self.makedirs(self._parent(path), exist_ok=True)
  File "/opt/anaconda3/lib/python3.7/site-packages/fsspec/implementations/local.py", line 47, in makedirs
    os.makedirs(path, exist_ok=exist_ok)
  File "/opt/anaconda3/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
NotADirectoryError: [Errno 20] Not a directory: '/ProjectDirectory/data/08_reporting/report_mass.xlsx/2020-11-28T17.37.57.113Z'

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

  • Kedro version: 0.16.2
  • Python version used (python -V): Python 3.7.8
  • Operating system and version: macOS 10.15.7
@clstaudt clstaudt added the Issue: Bug Report 🐞 Bug that needs to be fixed label Nov 28, 2020
@clstaudt clstaudt changed the title <Title> Versioning datasets fails with NotADirectoryError Nov 28, 2020
@clstaudt
Copy link
Author

The cause seems to be that previously the dataset was unversioned and a file already existed. After clearing the output folders manually it works.

I still leave the ticket, because being able to switch a dataset from versioned to unversioned and back by just editing the catalog file could be something Kedro wants to support.

@deepyaman
Copy link
Member

I still leave the ticket, because being able to switch a dataset from versioned to unversioned and back by just editing the catalog file could be something Kedro wants to support.

Given the way versioning is implemented, this isn't possible (in a straightforward, intuitive manner, at least). That being said, this is a common issue users face, and it could be nice to check for an unversioned dataset and return a more meaningful error message in that case.

@yetudada
Copy link
Contributor

@deepyaman Is right on this one. We do recognise that this is a poor user experience though. We'll look into it. I'll create a ticket for our backlog to try revisit this one again.

@yetudada yetudada changed the title Versioning datasets fails with NotADirectoryError [KED-2315] Versioning datasets fails with NotADirectoryError Nov 30, 2020
pull bot pushed a commit to vishalbelsare/kedro that referenced this issue Apr 4, 2021
@stale
Copy link

stale bot commented Apr 12, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Apr 12, 2021
@stale stale bot closed this as completed Apr 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Bug Report 🐞 Bug that needs to be fixed
Projects
None yet
Development

No branches or pull requests

3 participants