Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix kwargs propagation in interfaces module #280

Merged
merged 18 commits into from
Mar 10, 2023

Conversation

vinisalazar
Copy link
Contributor

Hi,

This is something I caught while running the ds = e.to_xarray(decode_times=False) line in 02-extras.ipynb.

Summary of changes

  • Fix kwargs propagation for urlopen and third-party libraries
  • Improve docstrings
  • Add typing.Dict for type annotation

vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Nov 22, 2022
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Nov 23, 2022
  - Code review for ioos#280
  - Rename **kw variables to the name of each library
@ocefpaf ocefpaf added the GSoC22 label Nov 24, 2022
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
@vinisalazar vinisalazar requested a review from ocefpaf November 29, 2022 03:46
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Nov 29, 2022
  - Use Optional[Dict] for all third-party libraries kwargs
  - Code review for ioos#280

  Co-authored-by: Filipe <ocefpaf@gmail.com>
  Co-authored-by: Alex Kerney <abk@mac.com>
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Nov 30, 2022
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Nov 30, 2022
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Nov 30, 2022
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
erddapy/core/interfaces.py Outdated Show resolved Hide resolved
@vinisalazar
Copy link
Contributor Author

Thanks for the code review, please have a look at 7c49ef6 too, I was having this problem before and I think this may be a good fix?

ocefpaf pushed a commit to vinisalazar/erddapy that referenced this pull request Dec 8, 2022
ocefpaf pushed a commit to vinisalazar/erddapy that referenced this pull request Dec 8, 2022
  - Code review for ioos#280
  - Rename **kw variables to the name of each library
ocefpaf pushed a commit to vinisalazar/erddapy that referenced this pull request Dec 8, 2022
  - Use Optional[Dict] for all third-party libraries kwargs
  - Code review for ioos#280

  Co-authored-by: Filipe <ocefpaf@gmail.com>
  Co-authored-by: Alex Kerney <abk@mac.com>
ocefpaf pushed a commit to vinisalazar/erddapy that referenced this pull request Dec 8, 2022
ocefpaf pushed a commit to vinisalazar/erddapy that referenced this pull request Dec 8, 2022
ocefpaf pushed a commit to vinisalazar/erddapy that referenced this pull request Dec 8, 2022
@ocefpaf ocefpaf force-pushed the fix-interfaces-kwargs branch from 7c49ef6 to 0ee3c96 Compare December 8, 2022 14:51
@ocefpaf
Copy link
Member

ocefpaf commented Dec 8, 2022

@vinisalazar and @abkfenris I rebased this one and, hopefully, I did not screw it up! Do you mind taking a second look so we can merge this one?

@abkfenris
Copy link
Contributor

Hmm, I think the doc test failures we're seeing are the difference in call signatures between the various to_X() functions and methods.

erddapy/erddapy/erddapy.py

Lines 334 to 347 in f11ad16

def to_pandas(self, **kw):
"""Save a data request to a pandas.DataFrame.
Accepts any `pandas.read_csv` keyword arguments.
This method uses the .csvp [1] response as the default for simplicity,
please check ERDDAP's documentation for the other csv options available.
[1] Download a ISO-8859-1 .csv file with line 1: name (units). Times are ISO 8601 strings.
"""
response = kw.pop("response", "csvp")
url = self.get_download_url(response=response, **kw)
return to_pandas(url, **kw)

vs

https://github.com/vinisalazar/erddapy/blob/0ee3c962996dc911d1a0a39c85bc6318ca4b5eef/erddapy/erddapy.py#L334-L353

What is more worrying is that they aren't being caught by other tests. While the new method matches up better to the underlying functions it breaks things for folks currently using erddapy.ERDDAP. Either we need to accept a breaking change, or do some smart shuffling of **kwargs around between methods and functions.

@vinisalazar
Copy link
Contributor Author

vinisalazar commented Dec 12, 2022

Thanks @abkfenris for pointing that out.

I found one use case in gliderpy which would surely break.

Could this be a possible fix? It preserves the old ERDDAP.to_pandas method while supporting the new interfaces.to_pandas kwargs propagation:

    def to_pandas(
        self,
        **kw
    ):
        """Save a data request to a pandas.DataFrame.

        Accepts any `pandas.read_csv` keyword arguments, passed as a dictionary to pandas_kwargs.

        This method uses the .csvp [1] response as the default for simplicity,
        please check ERDDAP's documentation for the other csv options available.

        [1] Download a ISO-8859-1 .csv file with line 1: name (units). Times are ISO 8601 strings.

        """
        response = kw.pop("response", "csvp")
        url = self.get_download_url(response=response, **kw)
        return to_pandas(url, pandas_kwargs=dict(**kw))

Edit: added by 73128a7 and 4ce66d9.

@vinisalazar vinisalazar requested a review from ocefpaf December 12, 2022 04:50
@vinisalazar
Copy link
Contributor Author

vinisalazar commented Feb 20, 2023

I believe this should pass after #295 is merged

edit: I rebased this branch after #295 was merged.

vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Feb 22, 2023
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Feb 22, 2023
  - Code review for ioos#280
  - Rename **kw variables to the name of each library
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Feb 22, 2023
  - Use Optional[Dict] for all third-party libraries kwargs
  - Code review for ioos#280

  Co-authored-by: Filipe <ocefpaf@gmail.com>
  Co-authored-by: Alex Kerney <abk@mac.com>
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Feb 22, 2023
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Feb 22, 2023
@vinisalazar vinisalazar force-pushed the fix-interfaces-kwargs branch from 4ce66d9 to 5892908 Compare February 22, 2023 03:17
vinisalazar added a commit to vinisalazar/erddapy that referenced this pull request Feb 22, 2023
@vinisalazar
Copy link
Contributor Author

@ocefpaf we are getting a ReadTimeout error when building the docs... any clue on why this is happening/what could be done?

vinisalazar and others added 18 commits March 10, 2023 22:05
  - Fix kwargs propagation for urlopen and third-party libraries
  - Improve docstrings
  - Add typing.Dict for type annotation
  - Code review for ioos#280
  - Rename **kw variables to the name of each library
  - Remove auth as positional arg for _nc_dataset method
  - Pass requests_kwargs dict in ERDDAP.to_xarray method
  - Remove kw.pop('auth') calls
Co-authored-by: Filipe <ocefpaf@gmail.com>
  - Use Optional[Dict] for all third-party libraries kwargs
  - Code review for ioos#280

  Co-authored-by: Filipe <ocefpaf@gmail.com>
  Co-authored-by: Alex Kerney <abk@mac.com>
  Adjust tests accordingly after the previous commit.
  - Remove statements initialising empty dict for requests_kwargs
  - Modify arguments in urlopen function call
  - Don't use conditional statements to create <third_party_lib>_kwargs dictionary
  This ensures backwards-compatibility with the ERDDAP class.

  - Use old ERDDAP.to_pandas(**kw) behaviour
  - Revert 90389be which modified test_to_objects
  - Use correct 'dataset_id' in 01a-griddap.ipynb
  - Convert longitude to required format
  - Use ERDDAP Sensors Gold Standard server
  - Change constraints and dataset_id
  - Same as 73128a7, but for xarray, iris, ncCF
@vinisalazar vinisalazar force-pushed the fix-interfaces-kwargs branch from ef26473 to 8322e67 Compare March 10, 2023 11:05
@ocefpaf
Copy link
Member

ocefpaf commented Mar 10, 2023

I'll try to fix the docs later today. Let's get this one in! Sorry for the delay Vini!

@ocefpaf ocefpaf merged commit c97cb91 into ioos:main Mar 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants