Skip to content

Fix handling of scalar date-time columns in iris.pandas cubes conversion #6955

@hsteptoe

Description

@hsteptoe

🐛 Bug Report

iris.pandas._get_dimensional_metadata requires improvements to test-coverage and code to handle scalar date-time columns during the cube conversion process.

This was noticed during #6948:

@trexfeathers Would appreciate your help in tracking down this error that relates to this call failing at https://github.com/hsteptoe/iris/blob/fc297b6ef463f3492832339b14ce691a03ddc710/lib/iris/pandas.py#L69.

Maybe I have typed values incorrectly, but it seems a possible bug that values can be passed something that isn't iterable??

The original code did indeed contain a bug, and you've made a mistake too.

Original bug

This code would never work when presented with a scalar time column, for multiple reasons (below). We never noticed because there simply isn't test coverage for scalar time columns, only scalar numeric columns.

Checking values.dtype

values will not always have a dtype when it is scalar, due to how NumPy handles object arrays (such as arrays of date-times`):

array.dtype array[0] hasattr(array[0], "dtype")
recognised e.g. float32/bool NumPy scalar e.g. np.float32(2.5) True
unknown i.e. object the original object e.g. cftime.datetime(... False

Iterating when values is any kind of date-time

You can't iterate through a single date-time object!

Fix

The easy part

Switch back to values.dtype == object and the test will pass again!

The hard part

We should probably have test coverage and correct handling of scalar date-time columns. I'd be happy with this being a separate issue, unless you wanted to go for extra credit!

Originally posted by @trexfeathers in #6948 (comment)

Expected behaviour

  • Add test to cover situation where iris.pandas.as_cubes is passed a pandas DataFrame with a scalar time column.
  • Add handling of scalar (non-iterable) time column to iris.pandas._get_dimensional_metadata.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions