Resolve serialization for numpy bool 1.x and 2.x compatibility

[sjyangkevin]

Create this PR to help review the solution for #52753
Closes: #52753

Unit Test

the testing ran in the breeze environment with numpy 1.26.4 installed. The following test case failed because it attempted to import numpy.bool. np.bool was a deprecated alias for the built-in Python bool type, introduced in NumPy version 1.20.0. np.bool_ is the Numpy scalar. If we keep both "numpy.bool" and "numpy.bool_" in the list of serializers and the list of deserializers, we might need to update this test case to run based on the installed numpy version. Or, update the code to check which numpy version we have and import and check the right bool (#52753 (comment)).

breeze testing core-tests --test-type Serialization

then, I created a init file files/airflow-breeze-config/init.sh to run uv pip install --upgrade pandas. This upgrade both pandas and numpy to the following versions. We may not only update one package, because the pandas depends on the numpy package and can result in the error mentioned #52753 (comment). The test case that failed when numpy 1.26.4 is installed now pass because numpy.bool is the NumPy scalar in 2.3.1.

Installed 2 packages in 116ms
- numpy==1.26.4
+ numpy==2.3.1
- pandas==2.1.4
+ pandas==2.3.1

If we only upgrade numpy to 2.x and didn't update pandas, the following error is raised. (uv pip install --upgrade numpy)

Installed 1 package in 188ms
- numpy==1.26.4
+ numpy==2.3.1

DAG Test (Run-time Behavior)

Numpy version 1.26.4

Numpy version 2.3.1

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

What fails exactly?

See the failing test case.

[potiuk]

[sjyangkevin]

Use xfail to capture the expected failure and show the documentation when np.bool is imported with NumPy 1.x installed.

[sjyangkevin]

Use xfail to capture the expected failure and show the documentation when np.bool is imported with NumPy 1.x installed.

Just found out the rest of the code will not be executed by using pytest.xfail in this way. Will think about another way to do it.

[ashb]

Are there any similar cases we need to account for with float or int between numpy 1.x and 2.x?

[sjyangkevin]

Are there any similar cases we need to account for with float or int between numpy 1.x and 2.x?

I have a test DAG #52753 (comment) which check for the integer types and the floating point types listed in the serializers. I think it is good with those types.

However, as mentioned here #53690 (comment). The np.float32 is not in the serializers list. So, it is likely that when this type is passed through, there is no serializer can be found for this. I can also run a test for it. (But it is a different thing from the issue we discussed)

[sjyangkevin]

I still need some time to get back to this, due to limited bandwidth this week. Will likely follow up again on this with updates early next week.

[potiuk]

I linked #54097 issue that is very much related (about float serialization) - I think the PR should attempt to address both as they are very related.

[sjyangkevin]

Hi @potiuk , @bolkedebruin , @uranusjr , and @ashb , thank you very much for your time and patient in reviewing this PR. I've pushed some small updates to see if it could resolve the concerns in our discussion.

Updates to numpy.py

1. The import_string is removed from numpy.py and the redundant check in the deserializer method is removed.
2. hasattr(np, "bool") and isinstance(o, np.bool) or isinstance(o, np.bool_) is used as a version compatible check for both numpy.bool_ (which is used as the NumPy scalar in NumPy version 1.x) and numpy.bool (which is used as the NumPy scalar in NumPy version 2.x, and numpy.bool_ becomes an alias of numpy.bool)
3. np.float32 is added to the list of serializers (i.e., serializers), as mentioned in Resolve serialization for numpy bool 1.x and 2.x compatibility #53690 (comment) and Resolve serialization for numpy bool 1.x and 2.x compatibility #53690 (comment).

More on the third bullet point, if the numpy.float32 is not in the list of serializers, we will get the following error. However, since it is part of the check if isinstance(o, (np.float16, np.float32, np.float64, np.complex64, np.complex128)):, probably make sense to have it in the list.

Test Cases

@uranusjr , I would like to elaborate more on the modifications to test_serializers_importable_and_str to see if it make sense to generate those test cases dynamically.

airflow/airflow-core/tests/unit/serialization/test_serde.py

Lines 389 to 411 in 9b1c16c

	def test_serializers_importable_and_str(self):
	"""Test if all distributed serializers are lazy loading and can be imported"""
	import airflow.serialization.serializers
	for _, name, _ in iter_namespace(airflow.serialization.serializers):
	if name == "airflow.serialization.serializers.iceberg":
	try:
	import pyiceberg # noqa: F401
	except ImportError:
	continue
	if name == "airflow.serialization.serializers.deltalake":
	try:
	import deltalake # noqa: F401
	except ImportError:
	continue
	mod = import_module(name)
	for s in getattr(mod, "serializers", list()):
	if not isinstance(s, str):
	raise TypeError(f"{s} is not of type str. This is required for lazy loading")
	try:
	import_string(s)
	except ImportError:
	raise AttributeError(f"{s} cannot be imported (located in {name})")

The above code is the current implementation in main branch. It is a single test case with some "special handles" for either compatibility considerations or optional dependencies. Basically, I think the test attempts to simulate the process of registering serializers and check if each serializer is importable.

I modified the test case by making it "parameterized" and the parameters are generated by the function generate_serializers_importable_tests. Therefore, for each serializer, a test case will be created, and we can have a control on each individual test from the test generation process, such as skipping some test case due to compatibility issue.

I can then run the test in breeze environment with different version of NumPy installed.

with NumPy 1.x installed. The numpy.bool test will be skipped because it is deprecated in NumPy 1.x.

with NumPy 2.x installed. Both numpy.bool and numpy.bool_ test cases will be executed because both are valid.

When a new serializer is added, this function can also capture those automatically to test for import, so it is less likely to miss any serializer. I also try to improve the documentation to see if it could help to provide more clarity. But, I am open for further feedback you have regarding this implementation.

Issue related to #54097

Looks like the issue is resolved through #54139. @potiuk please let me know if there is anything related to the issue that is still needed to be resolved here.

Thanks.

[sjyangkevin]

Not sure if it will be helpful to have "all version" and "full test" here to enable the checks for the canary build and compatibility kind of tests.

good point. Added/rebased to trigger it.

Thanks! Looking into the CI failure, it doesn’t look like related to the changes here. Will have another look into it later today.

[potiuk]

@bolkedebruin ?

[bolkedebruin]

checking

[bolkedebruin]

Ok I like it. The change in code is now small, tests cases are updated. To an extend I wonder if we are over engineering this (how many people still use numpy 1.X?)

[potiuk]

(how many people still use numpy 1.X?)

Precisely those people who will complain loudest :)

[bolkedebruin]

Faster than me in merging. Geez @potiuk :-P

[github-actions]

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 5babf15 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

[potiuk]

Faster than me in merging. Geez @potiuk :-P

Always watching 👀

[potiuk]

Backport attempt in #54881

[sjyangkevin]

Thanks and very appreciate your time in review.

I would like to learn more about the back porting. Is the reason we are doing this because we want to make the changes available in 3.0.x since the main branch is targeting 3.1.0? In this case, I think we need to backport the changes in another PR that updates the serde.py and the function signature from classname to class. Probably it is the main reason for the failed test in backport. Thanks.

[potiuk]

Thanks and very appreciate your time in review.

I would like to learn more about the back porting. Is the reason we are doing this because we want to make the changes available in 3.0.x since the main branch is targeting 3.1.0? In this case, I think we need to backport the changes in another PR that updates the serde.py and the function signature from classname to class. Probably it is the main reason for the failed test in backport. Thanks.

Yeah. I saw it already and there were conflicts, so I gave up on backporting. It's not worth it if backporting is somewhat complicated and issue is not too serious (and it has certain risks when backporting requires some manual changes). So... no backporting this time

[sjyangkevin]

Yeah. I saw it already and there were conflicts, so I gave up on backporting. It's not worth it if backporting is somewhat complicated and issue is not too serious (and it has certain risks when backporting requires some manual changes). So... no backporting this time

Got it, thanks for the feedback!