[Bug] protoc-wheel-0 dev dependency causing issues when building on musl-based distros like Alpine

[bryanlalexander]

What are you really trying to do?

I'm attempting to install pip install temporalio in Alpine v3.18 (Python 3.11.6). This doesn't seem to be possible given protoc-wheel-0 does not yet support newer versions (locked at 3.9). However, it does seem a minimal dependency that could be easily be isolated and eliminated.

Is this on the roadmap for removal/deprecation?

Describe the bug

ERROR: Could not find a version that satisfies the requirement protoc-wheel-0 (from versions: none)
ERROR: No matching distribution found for protoc-wheel-0

Collecting temporalio (from -r requirements.txt (line 1))
  Downloading temporalio-1.3.0.tar.gz (944 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 944.6/944.6 kB 19.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Collecting poetry-core>=1.0.0
        Obtaining dependency information for poetry-core>=1.0.0 from https://files.pythonhosted.org/packages/99/bc/058b8ff87871fce6615ad032d62c773272f243266b110f7b86d146cf78d8/poetry_core-1.8.1-py3-none-any.whl.metadata
        Downloading poetry_core-1.8.1-py3-none-any.whl.metadata (3.5 kB)
      Collecting setuptools
        Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
        Downloading setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
      Collecting wheel
        Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/fa/7f/4c07234086edbce4a0a446209dc0cb08a19bb206a3ea53b2f56a403f983b/wheel-0.41.3-py3-none-any.whl.metadata
        Downloading wheel-0.41.3-py3-none-any.whl.metadata (2.2 kB)
      Collecting setuptools-rust
        Obtaining dependency information for setuptools-rust from https://files.pythonhosted.org/packages/e0/34/d88a7ceb193fbcee6c8992d1b1e33ed20361027e07fea1676efc45ec7a43/setuptools_rust-1.8.1-py3-none-any.whl.metadata
        Downloading setuptools_rust-1.8.1-py3-none-any.whl.metadata (9.4 kB)
      ERROR: Could not find a version that satisfies the requirement protoc-wheel-0 (from versions: none)
      ERROR: No matching distribution found for protoc-wheel-0
      
Notice:       [notice] A new release of pip is available: 23.2.1 -> 23.3.1
Notice:       [notice] To update, run: pip install --upgrade pip
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Notice:  A new release of pip is available: 23.2.1 -> 23.3.1
Notice:  To update, run: pip install --upgrade pip

Minimal Reproduction

repro.dockerfile

FROM public.ecr.aws/docker/library/alpine:latest

RUN apk update && apk add --no-cache python3 py3-pip

RUN pip3 install temporalio

docker build -f repro.dockerfile .

Environment/Versions

program	version
Alpine	3.18
Python	3.11.6
pip	23.1.2

[cretz]

What happens on musl-based platforms is that your system attempts to rebuild the wheel from scratch. We don't expect runtime users to have this, it is a dev dependency. You're also going to have to have Rust and protoc on the path and everything else that comes with building this project since we don't pre-build a musl wheel.

We do require this dependency for proto generation to put protoc on the PATH, but I'm ok requiring it on the PATH already since we do for building the Rust part anyways. So we can probably remove this dev dependency (or make it optional), but you're still going to have to have certain things installed when building on an environment we don't prebuild a wheel for.

[bryanlalexander]

This seems to be perfectly reasonable to me, this will give those without a choice a path forward. I will move this job onto a debian-based distro for now. Will musl be supported in the future with this sdk?

[cretz]

I am unsure whether it will be supported in the future, but for now (as with windows arm, 32-bit, etc) it requires a custom build

[bryanlalexander]

Sounds good, thanks for the help!

[Carnewal]

So we can probably remove this dev dependency (or make it optional), but you're still going to have to have certain things installed when building on an environment we don't prebuild a wheel for.

I'm running into a similar issue, do you have any news with regards to this, any way to circumvent this?

 > [service 7/8] RUN poetry install --without dev:                                                                                                                                                        
1.643 Skipping virtualenv creation, as specified in config file.                                                                                                                                                               
2.530 Installing dependencies from lock file                                                                                                                                                                                   
3.263                                                                                                                                                                                                                          
3.264 Package operations: 38 installs, 0 updates, 0 removals                                                                                                                                                                   
3.264 
...
...
25.12   - Installing temporalio (1.6.0)
25.19   - Installing uvloop (0.19.0)
32.92 
32.92   ChefInstallError
32.92 
32.92   Failed to install poetry-core>=1.0.0, setuptools, wheel, setuptools-rust, protoc-wheel-0.
32.92   
32.92   Output:
32.92   Updating dependencies
32.92   Resolving dependencies...
32.92   
32.92   Package operations: 6 installs, 0 updates, 0 removals
32.92   
32.92     - Installing semantic-version (2.10.0)
32.92     - Installing setuptools (69.5.1)
32.92     - Installing poetry-core (1.9.0)
32.92     - Installing protoc-wheel-0 (25.3)
32.92     - Installing setuptools-rust (1.9.0)
32.92     - Installing wheel (0.43.0)
32.92   
32.92     RuntimeError
32.92   
32.92     Unable to find installation candidates for protoc-wheel-0 (25.3)
32.92   
32.92     at /usr/local/lib/python3.12/site-packages/poetry/installation/chooser.py:74 in choose_for
32.92          70│ 
32.92          71│             links.append(link)
32.92          72│ 
32.92          73│         if not links:
32.92       →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
32.92          75│ 
32.92          76│         # Get the best link
32.92          77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
32.92          78│ 
32.92   
32.92   Cannot install protoc-wheel-0.
32.92   
32.92   
32.92   
32.92   Error:
32.92   
32.93 
32.93   at /usr/local/lib/python3.12/site-packages/poetry/installation/chef.py:102 in install
32.95        98│             InstalledRepository.load(self._env),
32.95        99│         )
32.95       100│         installer.update(True)
32.95       101│         if installer.run() != 0:
32.95     → 102│             raise ChefInstallError(requirements, io.fetch_output(), io.fetch_error())
32.95       103│ 
32.95       104│ 
32.95       105│ class Chef:
32.95       106│     def __init__(
32.95 
32.95 Cannot install build-system.requires for temporalio.
32.95 
------

Moving to python-slim images removes the error but alpine would be preferred

[cretz]

I think this happens because we don't have a prebuilt musl wheel and protoc-wheel-0 doesn't either. I am not aware of a way to circumvent this at this time short of removing the dev dependency before building. I am reopening this issue to see if it's reasonable for us to remove this dependency (e.g. maybe put it behind a "extra" or use protoc on the PATH more directly).

[ymandge]

I have same issue when I try to install Temporal SDK on FreeBSD

`# python -m pip install temporalio
Collecting temporalio
Using cached temporalio-1.6.0.tar.gz (1.0 MB)
Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Collecting poetry-core>=1.0.0
Using cached poetry_core-1.9.0-py3-none-any.whl.metadata (3.5 kB)
Collecting setuptools
Using cached setuptools-70.0.0-py3-none-any.whl.metadata (5.9 kB)
Collecting wheel
Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Collecting setuptools-rust
Using cached setuptools_rust-1.9.0-py3-none-any.whl.metadata (9.3 kB)
ERROR: Could not find a version that satisfies the requirement protoc-wheel-0 (from versions: none)
ERROR: No matching distribution found for protoc-wheel-0
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.`

[cretz]

You can probably remove the

sdk-python/pyproject.toml

Line 50 in afadc15

protoc-wheel-0 = "^21.1"

dependency and rebuild. We need to remove this dependency ourselves or make it optional (we will look into doing this). It's only for letting us do proto generation which is not needed at build time.

[ymandge]

Hi @cretz,

After removing the above line, I was successfully able to build the dependencies using "poetry install --no-root --all-extras" command But when I try to build the release build using "poetry build" command , I'm getting the following errors,

`

poetry build

Preparing build environment with build-system requirements poetry-core>=1.0.0, setuptools, wheel, setuptools-rust, protoc-wheel-0

Command ['/tmp/tmpfe6rso36/.venv/bin/python', '/usr/local/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-24.0-py3-none-any.whl/pip', 'install', '--disable-pip-version-check', '--ignore-installed', '--no-input', 'poetry-core>=1.0.0', 'setuptools', 'wheel', 'setuptools-rust', 'protoc-wheel-0'] errored with the following return code 1

Output:
Collecting poetry-core>=1.0.0
Using cached poetry_core-1.9.0-py3-none-any.whl.metadata (3.5 kB)
Collecting setuptools
Using cached setuptools-70.0.0-py3-none-any.whl.metadata (5.9 kB)
Collecting wheel
Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Collecting setuptools-rust
Using cached setuptools_rust-1.9.0-py3-none-any.whl.metadata (9.3 kB)
ERROR: Could not find a version that satisfies the requirement protoc-wheel-0 (from versions: none)
ERROR: No matching distribution found for protoc-wheel-0
`

Fyi - I'm referring below doc

https://github.com/temporalio/sdk-python?tab=readme-ov-file#development:~:text=written%20in%20Rust.-,Building,-Prepare

for building the Temporal Python SDK on FreeBSD.

[cretz]

To remove a dependency with Poetry, you it needs to be removed from toml and lock file. poetry remove command can probably do this.

[ymandge]

Hey everyone, fyi, I was able to build and install the Temporal Python SDK on FreeBSD by removing the protoc-wheel-0 and adding the protobuf package. Thank you Temporal Team for your prompt help :)

#protoc-wheel-0 = "^21.1" ---> remove this line
protobuf = "^4.25.3" ---> add this one

[hansraj1999]

Using cached temporalio-1.7.0.tar.gz (1.3 MB)
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Collecting poetry-core>=1.0.0
Using cached poetry_core-1.9.0-py3-none-any.whl.metadata (3.5 kB)
Collecting setuptools
Using cached setuptools-73.0.1-py3-none-any.whl.metadata (6.6 kB)
Collecting wheel
Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
Collecting setuptools-rust
Using cached setuptools_rust-1.10.1-py3-none-any.whl.metadata (9.2 kB)
ERROR: Could not find a version that satisfies the requirement protoc-wheel-0 (from versions: none)
ERROR: No matching distribution found for protoc-wheel-0
end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

base image python 3.11-alpine3.16
getting this error while running python -m pip install temporalio command

[cretz]

I see an issue was opened at #622 and a new Slack thread was started. No need to double/triple post.