Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: publish to conda #157

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

ci: publish to conda #157

wants to merge 4 commits into from

Conversation

bonjourmauko
Copy link
Member

@bonjourmauko bonjourmauko commented Oct 15, 2024 •
edited
Loading

Being blocked by openfisca/openfisca-core#1281
Depends on #159
Related to #124

TODO

  • Add instructions to build conda locally
  • Build conda
  • Test conda
  • Deploy

Build

  • Build, test, and publish to conda

@bonjourmauko bonjourmauko force-pushed the ci/publish-to-conda-v2 branch 8 times, most recently from 3b4afde to 0bb462e Compare October 15, 2024 14:36
@bonjourmauko bonjourmauko mentioned this pull request Oct 15, 2024
Open
@bonjourmauko bonjourmauko added the kind:build Changes that affect the build system or external dependencies label Oct 15, 2024
@bonjourmauko bonjourmauko self-assigned this Oct 15, 2024
@bonjourmauko bonjourmauko changed the base branch from main to build/normalize-pyproject October 15, 2024 14:56
@bonjourmauko bonjourmauko mentioned this pull request Oct 15, 2024
Open
Base automatically changed from build/normalize-pyproject to main October 16, 2024 13:24
@bonjourmauko bonjourmauko changed the base branch from main to ci/test-built October 16, 2024 13:34
@benoit-cty
Copy link
Contributor

Can you explain how you run rattler-build locally ?
I try:

docker run --volume $PWD:/openfisca -i -t continuumio/anaconda3 /bin/bash
cd /openfisca
conda install -c conda-forge rattler-build
rattler-build build -r .conda/recipe.yaml

But it ends up with:

  ├─▶ failed to create directory `/openfisca/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/
  │   bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-
  │   template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/
  │   rattler-build_openfisca-country-template_1729244799/work/.conda`
  ╰─▶ File name too long (os error 36)

Base automatically changed from ci/test-built to main October 18, 2024 10:14
@bonjourmauko
Copy link
Member Author

Can you explain how you run rattler-build locally ? I try:

docker run --volume $PWD:/openfisca -i -t continuumio/anaconda3 /bin/bash
cd /openfisca
conda install -c conda-forge rattler-build
rattler-build build -r .conda/recipe.yaml

But it ends up with:

  ├─▶ failed to create directory `/openfisca/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/
  │   output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-
  │   country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/
  │   bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-
  │   template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/rattler-build_openfisca-country-template_1729244799/work/output/bld/
  │   rattler-build_openfisca-country-template_1729244799/work/.conda`
  ╰─▶ File name too long (os error 36)

The error you're getting is because of the output-dir path.

I'm passing the --output-dir /tmp/conda-bld option to rattler-build to fix it.

@benoit-cty
Copy link
Contributor

Sorry,
I have the same problem File name too long (os error 36) using Conda docker image or Pixi docker image and running : rattler-build build --output-dir /tmp/conda-bld -r .conda/recipe.yaml

How do you install rattler-build ?

@bonjourmauko
Copy link
Member Author

Sorry, I have the same problem File name too long (os error 36) using Conda docker image or Pixi docker image and running : rattler-build build --output-dir /tmp/conda-bld -r .conda/recipe.yaml

How do you install rattler-build ?

In local, with 'pixi', and running the same as in core:

rattler-build build --channel openfisca --channel conda-forge --recipe .conda --output-dir ~/Desktop/asdf

@benoit-cty
Copy link
Contributor

I'm sorry but, I install pixi and rattler-build on my native Ubuntu 24.04 and still have the same error Error: × FileSystem error: 'File name too long (os error 36)'

country-template on  ci/publish-to-conda-v2 [?] is 📦 v8.0.0 via 🐍 v3.9.20 
❯ getconf NAME_MAX /dev/nvme0n1p1
255

country-template on  ci/publish-to-conda-v2 [?] is 📦 v8.0.0 via 🐍 v3.9.20 
❯ getconf PATH_MAX /dev/nvme0n1p1
4096

@bonjourmauko
Copy link
Member Author

I'm sorry but, I install pixi and rattler-build on my native Ubuntu 24.04 and still have the same error Error: × FileSystem error: 'File name too long (os error 36)'

country-template on  ci/publish-to-conda-v2 [?] is 📦 v8.0.0 via 🐍 v3.9.20 
❯ getconf NAME_MAX /dev/nvme0n1p1
255

country-template on  ci/publish-to-conda-v2 [?] is 📦 v8.0.0 via 🐍 v3.9.20 
❯ getconf PATH_MAX /dev/nvme0n1p1
4096

Ok. I'll start by adding instructions to the README.md on how to build this with rattler locally. If it still does not work for you, we can move back to conda-build. I'll ping you when it is done.

@benoit-cty
Copy link
Contributor

benoit-cty commented Oct 21, 2024 •
edited
Loading

Surprisingly , on my laptop running Ubuntu 20.04.6 LTS it's working:

curl -fsSL https://pixi.sh/install.sh | bash
source /home/ben/.bashrc
pixi global install rattler-build
rattler-build build --channel openfisca --channel conda-forge --recipe .conda --output-dir /tmp/rattler

Despite having the same PATH_MAX and NAME_MAX.

@bonjourmauko
Copy link
Member Author

Surprisingly , on my laptop running Ubuntu 20.04.6 LTS it's working:

curl -fsSL https://pixi.sh/install.sh | bash
source /home/ben/.bashrc
pixi global install rattler-build
rattler-build build --channel openfisca --channel conda-forge --recipe .conda --output-dir /tmp/rattler

Despite having the same PATH_MAX and NAME_MAX.

I liked rattler mostly because I was putting to much time with conda and its 1000 bugs.

And it is faster! I hope you can make it work, I'd really like to use this one generally.

@bonjourmauko
Copy link
Member Author

Concerning the failure in the tests part, I don't know if the conda build fails as well. If that is so, then there is a problem directly in core. I hope that with the poetry changes we will see clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@bonjourmauko bonjourmauko

Labels
kind:build Changes that affect the build system or external dependencies
Projects
OpenFisca International
Status: Inbox
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bonjourmauko @benoit-cty