GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 #44395

kou · 2024-10-13T22:24:50Z

Rationale for this change

Because JPype doesn't support Python 3.13 yet.

What changes are included in this PR?

Pin Python 3.12.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [Java][Integration][Release] verify-rc-source-integration-linux-conda-latest-amd64 fails with MemoryError #44389

…fy-rc-source-integration-linux-conda-latest-amd64 Because JPype doesn't support Python 3.13 yet. See also: * apacheGH-44389 * https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5

github-actions · 2024-10-13T22:25:18Z

⚠️ GitHub issue #44389 has been automatically assigned in GitHub to PR creator.

kou · 2024-10-13T22:25:30Z

@github-actions crossbow submit verify-rc-source--linux-

github-actions · 2024-10-13T22:28:01Z

Revision: 8a9a6be

Submitted crossbow builds: ursacomputing/crossbow @ actions-b9724982b4

Task	Status
verify-rc-source-cpp-linux-almalinux-8-amd64
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-cpp-linux-ubuntu-20.04-amd64
verify-rc-source-cpp-linux-ubuntu-22.04-amd64
verify-rc-source-csharp-linux-almalinux-8-amd64
verify-rc-source-csharp-linux-conda-latest-amd64
verify-rc-source-csharp-linux-ubuntu-20.04-amd64
verify-rc-source-csharp-linux-ubuntu-22.04-amd64
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-js-linux-almalinux-8-amd64
verify-rc-source-js-linux-conda-latest-amd64
verify-rc-source-js-linux-ubuntu-20.04-amd64
verify-rc-source-js-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-ruby-linux-almalinux-8-amd64
verify-rc-source-ruby-linux-conda-latest-amd64
verify-rc-source-ruby-linux-ubuntu-20.04-amd64
verify-rc-source-ruby-linux-ubuntu-22.04-amd64

lidavidm · 2024-10-14T00:13:30Z

It seems 3.13 still got installed?

dev/tasks/verify-rc/github.linux.amd64.yml

…nux.amd64.yml

raulcd · 2024-10-14T10:53:21Z

@github-actions crossbow submit verify-rc-source--linux-

github-actions · 2024-10-14T10:55:54Z

Revision: 158c136

Submitted crossbow builds: ursacomputing/crossbow @ actions-15b8e1c9d8

Task	Status
verify-rc-source-cpp-linux-almalinux-8-amd64
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-cpp-linux-ubuntu-20.04-amd64
verify-rc-source-cpp-linux-ubuntu-22.04-amd64
verify-rc-source-csharp-linux-almalinux-8-amd64
verify-rc-source-csharp-linux-conda-latest-amd64
verify-rc-source-csharp-linux-ubuntu-20.04-amd64
verify-rc-source-csharp-linux-ubuntu-22.04-amd64
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-js-linux-almalinux-8-amd64
verify-rc-source-js-linux-conda-latest-amd64
verify-rc-source-js-linux-ubuntu-20.04-amd64
verify-rc-source-js-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-ruby-linux-almalinux-8-amd64
verify-rc-source-ruby-linux-conda-latest-amd64
verify-rc-source-ruby-linux-ubuntu-20.04-amd64
verify-rc-source-ruby-linux-ubuntu-22.04-amd64

raulcd · 2024-10-14T11:26:03Z

@jorisvandenbossche any idea why the Python verification has just started failing for the above?

=================================== FAILURES ===================================
_____________________________ test_legacy_int_type _____________________________

    def test_legacy_int_type():
        with pytest.warns(FutureWarning, match="PyExtensionType is deprecated"):
            ext_ty = LegacyIntType()
        arr = pa.array([1, 2, 3], type=ext_ty.storage_type)
        ext_arr = pa.ExtensionArray.from_storage(ext_ty, arr)
        batch = pa.RecordBatch.from_arrays([ext_arr], names=['ext'])
        buf = ipc_write_batch(batch)
    
        with pytest.warns((RuntimeWarning, FutureWarning)):
>           batch = ipc_read_batch(buf)

pyarrow/tests/test_extension_type.py:1722: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/tests/test_extension_type.py:254: in ipc_read_batch
    reader = pa.RecordBatchStreamReader(buf)
pyarrow/ipc.py:52: in __init__
    self._open(source, options=options, memory_pool=memory_pool)
pyarrow/ipc.pxi:1006: in pyarrow.lib._RecordBatchStreamReader._open
    self.reader = GetResultValue(CRecordBatchStreamReader.Open(
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
    return check_status(status)
pyarrow/error.pxi:89: in pyarrow.lib.check_status
    RestorePyError(status)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'pandas.core.arrays.arrow.extension_types.patch_pyarrow.<locals>.ForbiddenExtensionType'>
storage_type = DataType(int8)
serialized = b'\x80\x04\x95:\x00\x00\x00\x00\x00\x00\x00\x8c!pyarrow.tests.test_extension_type\x94\x8c\rLegacyIntType\x94\x93\x94)R\x94.'

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        import io
        import pickletools
    
        out = io.StringIO()
        pickletools.dis(serialized, out)
>       raise RuntimeError(
            _ERROR_MSG.format(
                storage_type=storage_type,
                serialized=serialized,
                pickle_disassembly=out.getvalue(),
            )
        )
E       RuntimeError: Disallowed deserialization of 'arrow.py_extension_type':
E       storage_type = int8
E       serialized = b'\x80\x04\x95:\x00\x00\x00\x00\x00\x00\x00\x8c!pyarrow.tests.test_extension_type\x94\x8c\rLegacyIntType\x94\x93\x94)R\x94.'
E       pickle disassembly:
E           0: \x80 PROTO      4
E           2: \x95 FRAME      58
E          11: \x8c SHORT_BINUNICODE 'pyarrow.tests.test_extension_type'
E          46: \x94 MEMOIZE    (as 0)
E          47: \x8c SHORT_BINUNICODE 'LegacyIntType'
E          62: \x94 MEMOIZE    (as 1)
E          63: \x93 STACK_GLOBAL
E          64: \x94 MEMOIZE    (as 2)
E          65: )    EMPTY_TUPLE
E          66: R    REDUCE
E          67: \x94 MEMOIZE    (as 3)
E          68: .    STOP
E       highest protocol among opcodes = 4
E       
E       
E       Reading of untrusted Parquet or Feather files with a PyExtensionType column
E       allows arbitrary code execution.
E       If you trust this file, you can enable reading the extension type by one of:
E       
E       - upgrading to pyarrow >= 14.0.1, and call `pa.PyExtensionType.set_auto_load(True)`
E       - install pyarrow-hotfix (`pip install pyarrow-hotfix`) and disable it by running
E         `import pyarrow_hotfix; pyarrow_hotfix.uninstall()`
E       
E       We strongly recommend updating your Parquet/Feather files to use extension types
E       derived from `pyarrow.ExtensionType` instead, and register this type explicitly.

/tmp/arrow-HEAD.LSNAi/venv-source/lib64/python3.11/site-packages/pandas/core/arrays/arrow/extension_types.py:158: RuntimeError

During handling of the above exception, another exception occurred:

    def test_legacy_int_type():
        with pytest.warns(FutureWarning, match="PyExtensionType is deprecated"):
            ext_ty = LegacyIntType()
        arr = pa.array([1, 2, 3], type=ext_ty.storage_type)
        ext_arr = pa.ExtensionArray.from_storage(ext_ty, arr)
        batch = pa.RecordBatch.from_arrays([ext_arr], names=['ext'])
        buf = ipc_write_batch(batch)
    
>       with pytest.warns((RuntimeWarning, FutureWarning)):
E       Failed: DID NOT WARN. No warnings of type (<class 'RuntimeWarning'>, <class 'FutureWarning'>) were emitted.
E        Emitted warnings: [].

raulcd · 2024-10-14T11:42:23Z

@github-actions crossbow submit verify-rc-source-integration-linux-conda-latest-amd64

github-actions · 2024-10-14T11:44:37Z

Revision: 22d2412

Submitted crossbow builds: ursacomputing/crossbow @ actions-795491bc3b

Task	Status
verify-rc-source-integration-linux-conda-latest-amd64

raulcd · 2024-10-14T13:02:11Z

@github-actions crossbow submit verify-rc-source-integration-linux-conda-latest-amd64

github-actions · 2024-10-14T13:04:24Z

Revision: b543a5b

Submitted crossbow builds: ursacomputing/crossbow @ actions-9e116399c9

Task	Status
verify-rc-source-integration-linux-conda-latest-amd64

raulcd · 2024-10-14T13:50:46Z

@github-actions crossbow submit verify-rc-source--linux-

github-actions · 2024-10-14T13:53:32Z

Revision: b543a5b

Submitted crossbow builds: ursacomputing/crossbow @ actions-e3f57c0557

Task	Status
verify-rc-source-cpp-linux-almalinux-8-amd64
verify-rc-source-cpp-linux-conda-latest-amd64
verify-rc-source-cpp-linux-ubuntu-20.04-amd64
verify-rc-source-cpp-linux-ubuntu-22.04-amd64
verify-rc-source-csharp-linux-almalinux-8-amd64
verify-rc-source-csharp-linux-conda-latest-amd64
verify-rc-source-csharp-linux-ubuntu-20.04-amd64
verify-rc-source-csharp-linux-ubuntu-22.04-amd64
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-js-linux-almalinux-8-amd64
verify-rc-source-js-linux-conda-latest-amd64
verify-rc-source-js-linux-ubuntu-20.04-amd64
verify-rc-source-js-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-ruby-linux-almalinux-8-amd64
verify-rc-source-ruby-linux-conda-latest-amd64
verify-rc-source-ruby-linux-ubuntu-20.04-amd64
verify-rc-source-ruby-linux-ubuntu-22.04-amd64

raulcd · 2024-10-14T13:55:53Z

These jobs are from main. Just validating status there

Revision: 27e56f0

Submitted crossbow builds: ursacomputing/crossbow @ verify-rc-source-python-main-1

Task	Status
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

raulcd · 2024-10-14T14:51:04Z

I am really confused, why is the almalinux job, for example, failing here but not on main, when the change here is not on the workflow for the job: https://github.com/ursacomputing/crossbow/actions/runs/11328978008/workflow

kou · 2024-10-16T05:49:49Z

The failed job (that uses my fork) creates pyarrow-13.0.0.dev3160+gb543a5b94:

https://github.com/ursacomputing/crossbow/actions/runs/11328978008/job/31503426291#step:6:9674

  Created wheel for pyarrow: filename=pyarrow-13.0.0.dev3160+gb543a5b94-0.editable-cp311-cp311-linux_x86_64.whl size=25237 sha256=71df89345437a7dcc15e66625293a68b09065ac4e48c3e76ef6ada73a2657bfb

But the succeeded job (that uses apache/arrow) creates pyarrow-18.0.0.dev454+g27e56f01e:

https://github.com/ursacomputing/crossbow/actions/runs/11329033094/job/31503611194#step:6:9730

  Created wheel for pyarrow: filename=pyarrow-18.0.0.dev454+g27e56f01e-0.editable-cp311-cp311-linux_x86_64.whl size=25215 sha256=acd41891d2d298bfb3c56011eaeaa80d8bfa9049bfe6a08418f14f7aab266a04

I think that it's caused by my fork doesn't have all apache-arrow-* tags. I've pushed tags (git push --tags origin) and restarted failed jobs.

raulcd

👍 Thanks @kou ! The tag issue makes sense. Thanks for fixing it. I've created RC0 already, I'll merge this for 19.0.0 and will add the backport-candidate label. If I have to create a new RC I'll add to 18.0.0 just to fix the CI failure.

conbench-apache-arrow · 2024-10-16T17:51:43Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit d0e7d07.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

kou requested review from assignUser, jonkeane and raulcd as code owners October 13, 2024 22:24

github-actions bot added the awaiting committer review Awaiting committer review label Oct 13, 2024

kou mentioned this pull request Oct 13, 2024

[Java][Integration][Release] verify-rc-source-integration-linux-conda-latest-amd64 fails with MemoryError #44389

Closed

raulcd reviewed Oct 14, 2024

View reviewed changes

dev/tasks/verify-rc/github.linux.amd64.yml Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 14, 2024

Check for conda on github.linux.amd64.docker.yml instead of github.li…

158c136

…nux.amd64.yml

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 14, 2024

Pass environment variable to archery command

22d2412

Fix template comment

b543a5b

This comment was marked as outdated.

Sign in to view

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 14, 2024

raulcd approved these changes Oct 16, 2024

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Oct 16, 2024

raulcd merged commit d0e7d07 into apache:main Oct 16, 2024
8 checks passed

raulcd removed the awaiting merge Awaiting merge label Oct 16, 2024

kou deleted the ci-integration-linux-jpype branch October 16, 2024 12:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 #44395

GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 #44395

kou commented Oct 13, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Oct 13, 2024

kou commented Oct 13, 2024

github-actions bot commented Oct 13, 2024

lidavidm commented Oct 14, 2024

raulcd commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024

raulcd commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024

This comment was marked as outdated.

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024 •

edited

Loading

raulcd commented Oct 14, 2024

kou commented Oct 16, 2024

raulcd left a comment

conbench-apache-arrow bot commented Oct 16, 2024

GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 #44395

GH-44389: [Java][Integration][Release] Use Python 3.12 for verify-rc-source-integration-linux-conda-latest-amd64 #44395

Conversation

kou commented Oct 13, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

github-actions bot commented Oct 13, 2024

kou commented Oct 13, 2024

github-actions bot commented Oct 13, 2024

lidavidm commented Oct 14, 2024

raulcd commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024

raulcd commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024

This comment was marked as outdated.

github-actions bot commented Oct 14, 2024

raulcd commented Oct 14, 2024 • edited Loading

These jobs are from main. Just validating status there

raulcd commented Oct 14, 2024

kou commented Oct 16, 2024

raulcd left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Oct 16, 2024

kou commented Oct 13, 2024 •

edited by github-actions bot

Loading

raulcd commented Oct 14, 2024 •

edited

Loading