Deployment on Airflow fails due to Dependencies

[crash-absolutelabs]

Hello! I've been trying to deploy this pipeline on GCP Composer and I hit pre-installed dependency issues.

I'll keep this thread updated as I work through all of them, but I'm attempting to get this pipeline deployed.

Specifically it starts with pre-installed google-ads requiring protobuf ~3.19.5 when >=3.20.0 is installed.

I'll update this thread once I find a set a of dependency versions that work :)

[crash-absolutelabs]

Alright, I have what seems to be a working set of dependencies installed! I haven't deployed the airflow DAGs yet, however I
'll be trying that soon.

Here are the dependency versions I landed on:

eth-hash=0.3.3
web3==5.29.2
eth-abi==2.1.1
eth-rlp==0.2.1
ethereum-etl

[charlielewisme]

I suggest if you have a working setup, then pin everything (including Composer version) and only update once in a while, since it would be annoying to have to revisit dependency conflicts just for small version bumps.

Composer is remarkably pedantic about conflicts in additional pypi packages. Many times I've had to manually tinker with dependency conflicts where I have had no issues when testing locally. When updating dependencies, I think it does something "cleverer" than just pip install -r requirements.txt -- it can be infuriating, and I agree with your findings that adding eth-hash and/or eth-rlp have sometimes unblocked me with ethereum-etl and its forks. In fact, that's exactly why eth-hash==0.3.3 is included in requirements_airflow.txt

Fwiw, our current production setup is:

• composer-2.0.25-airflow-2.2.5
• pypi packages pinned: eth-hash==0.3.3, ethereum-etl==2.0.5
  • I don't think the version of ethereum-etl should make any difference between 2.0.3 and current latest 2.1.1
  • Saying this, I might update requirements_airflow.txt for consistency

When I SSH into the airflow-worker and pip list, it includes:

eth-abi==2.1.1
eth-hash=0.3.3
eth-rlp==0.2.1
ethereum-etl==2.0.5
google-ads==18.0.0
protobuf==3.20.0
web3==5.30.0

So, apart from web3, very similar to yours in the end but we were fortunate in only having to add eth-hash, at least for ethereum-etl.

[crash-absolutelabs]

Yup, I think a lot of this could be solved if ethereum-etl could update to a newer version of eth-utils :-D
Thank you for confirming this is a common problem with composer as I thought maybe I was just going crazy!

I think for other new users it might be worth updating the readme with some of this information so others don't struggle as much as I did. If that's useful I'll happily create a PR for it :)

[charlielewisme]

Thanks for the suggestion about eth-utils. I can't prioritise this work right now, but I will come back to this and try to update the dependencies and improve this.

For now, feel free to raise a PR for updating the README and put me as a reviewer or assignee, happy to help here.