Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix - lock cython version, add missing pip deps #667

Merged
merged 3 commits into from
Nov 8, 2023

Conversation

JoshCu
Copy link
Contributor

@JoshCu JoshCu commented Oct 28, 2023

Fixes #651
breakout from #660

Summary

Adds required pip packages to requirements.txt
Locks cython version to fix ./compile.sh

requirements.txt

Additions

  • pyarrow
  • deprecated
  • geopandas

Changes

  • Cython==0.29.36

Testing

Before

#0 65.83 Compiler crash traceback from this point on:
#0 65.83   File "/opt/venv/lib64/python3.9/site-packages/Cython/Compiler/ExprNodes.py", line 4968, in analyse_types
#0 65.83     performance_hint(index.pos, "Index should be typed for more efficient access")
#0 65.83 TypeError: performance_hint() missing 1 required positional argument: 'env'
#0 65.83 Traceback (most recent call last):
#0 65.83   File "/t-route/src/troute-routing/setup.py", line 122, in <module>
#0 65.83 Fortran compiler type is: gnu95
#0 65.83 Compiling troute/routing/fast_reach/reach.pyx because it changed.
#0 65.83 Compiling troute/routing/fast_reach/mc_reach.pyx because it changed.
#0 65.83 Compiling troute/routing/fast_reach/diffusive.pyx because it changed.
#0 65.83 Compiling troute/routing/fast_reach/simple_da.pyx because it changed.
#0 65.83 [1/4] Cythonizing troute/routing/fast_reach/diffusive.pyx
#0 65.83 [2/4] Cythonizing troute/routing/fast_reach/mc_reach.pyx
#0 65.83     ext_modules = cythonize(ext_modules, compiler_directives={"language_level": 3})
#0 65.83   File "/opt/venv/lib64/python3.9/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
#0 65.83     cythonize_one(*args)
#0 65.83   File "/opt/venv/lib64/python3.9/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
#0 65.83     raise CompileError(None, pyx_file)
#0 65.83 Cython.Compiler.Errors.CompileError: troute/routing/fast_reach/mc_reach.pyx
------
Dockerfile.troute:20
--------------------
  18 |     RUN python -m pip install -r requirements.txt
  19 |
  20 | >>> RUN ./compiler.sh no-e
  21 |
  22 |     # increase max open files soft limit
--------------------
ERROR: failed to solve: process "/bin/sh -c ./compiler.sh no-e" did not complete successfully: exit code: 1

After

image
Screenshot is of a docker build step successfully running, before it would fail on that step.

Notes

Using legacy 'setup.py install' for troute.network, since package 'wheel' is not installed. this warning does appear when compiling.
Without version locks on all the pip packages, it would be a good idea to have some schedulled tests to catch any breaking updates to packages. #669 or #670 could help?

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

requirements.txt Outdated Show resolved Hide resolved
@JoshCu JoshCu marked this pull request as ready for review October 28, 2023 18:54
up from 0.29.36
@JoshCu
Copy link
Contributor Author

JoshCu commented Nov 2, 2023

As this is still open and related to this #614 I'm going to also set pyarrow <12.0 and retest

@JoshCu
Copy link
Contributor Author

JoshCu commented Nov 2, 2023

As this is still open and related to this #614 I'm going to also set pyarrow <12.0 and retest

image
build works as expected with both cython<3.0.4 and pyyarrow <12.0

pandas>=1.1.5
xarray
netcdf4
pyyaml
toolz
joblib
deprecated
pyarrow<12.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please double check the installation and test run without any version specification on thepyarrow? Because on my branch here build log, downloads pyarrow-14.0.0 and test ran just fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the pyarrow version is an issue specific to running t-route with ngen, #614
I'm not sure if it would be best to address the issue closest to the root (here) to stop other things that use it all needing the same patch. Or if it should be addressed in ngen as that's where it appears to break things?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JoshCu @TrupeshKumarPatel I tested these changes and they work fine on my end. Is the above conversation about locking in pyarrow version ongoing or is this PR ready for approval/merging?

@shorvath-noaa shorvath-noaa merged commit 8cf78b4 into NOAA-OWP:master Nov 8, 2023
@JoshCu JoshCu deleted the pip_package_fix branch November 10, 2023 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error in installation
4 participants