Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add extra clarification about fs_args #4112

Merged
merged 2 commits into from
Aug 22, 2024
Merged

Add extra clarification about fs_args #4112

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6104fb9
Add extra clarification about fs args
merelcht Aug 22, 2024
78be4a6
Fix spacing
merelcht Aug 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions docs/source/data/data_catalog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,10 @@

This section explains the additional settings available within `catalog.yml`.

### Load and save arguments
The Kedro Data Catalog also accepts two different groups of `*_args` parameters that serve different purposes:
### Load, save and filesystem arguments
The Kedro Data Catalog also accepts different groups of `*_args` parameters that serve different purposes:

* **`load_args` and `save_args`**: Configures how a third-party library loads/saves data from/to a file. In the spaceflights example above, `load_args`, is passed to the excel file read method (`pd.read_excel`) as a [keyword argument](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html). Although not specified here, the equivalent output is `save_args` and the value would be passed to [`pd.DataFrame.to_excel` method](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html).
* **`load_args` and `save_args`**: Configure how a third-party library loads/saves data from/to a file. In the spaceflights example above, `load_args`, is passed to the excel file read method (`pd.read_excel`) as a [keyword argument](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html). Although not specified here, the equivalent output is `save_args` and the value would be passed to [`pd.DataFrame.to_excel` method](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_excel.html).

Check warning on line 105 in docs/source/data/data_catalog.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/data/data_catalog.md#L105 

[Kedro.toowordy] 'equivalent' is too wordy
Raw output 
{"message": "[Kedro.toowordy] 'equivalent' is too wordy", "location": {"path": "docs/source/data/data_catalog.md", "range": {"start": {"line": 105, "column": 352}}}, "severity": "WARNING"}

For example, to load or save a CSV on a local file system, using specified load/save arguments:

Expand Down Expand Up @@ -143,6 +143,22 @@
encoding: "utf-8"
```

If you want to save a file in append mode instead of overwrite you can use the `open_args_save` `mode` parameter:

```yaml
test_dataset:
type: ...
fs_args:
open_args_save:
mode: "a"
```

```{note}
Default load, save and filesystem arguments are defined inside the specific dataset implementations as `DEFAULT_LOAD_ARGS`, `DEFAULT_SAVE_ARGS`, and `DEFAULT_FS_ARGS` respectively.
You can check those in {py:mod}`the dataset API documentation <kedro-datasets:kedro_datasets>`.
```


### Dataset access credentials
The Data Catalog also works with the `credentials.yml` file in `conf/local/`, allowing you to specify usernames and passwords required to load certain datasets.

Expand Down