Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: more structured toc #1142

Merged
merged 16 commits into from
Apr 9, 2024
Merged
ruben-arts marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,10 +1,7 @@
# Proposal Design: Multi Environment Support
## Objective
The aim is to introduce an environment set mechanism in the `pixi` package manager.
This mechanism will enable clear, conflict-free management of dependencies tailored to specific environments, while also maintaining the integrity of fixed lockfiles.

# Multi Environment Support

### Motivating Example

There are multiple scenarios where multiple environments are useful.

- **Testing of multiple package versions**, e.g. `py39` and `py310` or polars `0.12` and `0.13`.
Expand All @@ -17,17 +14,17 @@ There are multiple scenarios where multiple environments are useful.
This prepares `pixi` for use in large projects with multiple use-cases, multiple developers and different CI needs.

## Design Considerations

There are a few things we wanted to keep in mind in the desin:

tdejager marked this conversation as resolved.
Show resolved Hide resolved
1. **User-friendliness**: Pixi is a user focussed tool that goes beyond developers. The feature should have good error reporting and helpful documentation from the start.
2. **Keep it simple**: Not understanding the multiple environments feature shouldn't limit a user to use pixi. The feature should be "invisible" to the non-multi env use-cases.
3. **No Automatic Combinatorial**: To ensure the dependency resolution process remains manageable, the solution should avoid a combinatorial explosion of dependency sets. By making the environments user defined and not automatically inferred by testing a matrix of the features.
4. **Single environment Activation**: The design should allow only one environment to be active at any given time, simplifying the resolution process and preventing conflicts.
5. **Fixed Lockfiles**: It's crucial to preserve fixed lockfiles for consistency and predictability. Solutions must ensure reliability not just for authors but also for end-users, particularly at the time of lockfile creation.

## Proposed Solution
!!! important
This is a proposal, not a final design. The proposal is open for discussion and will be updated based on the feedback.

### Feature & Environment Set Definitions

Introduce environment sets into the `pixi.toml` this describes environments based on `feature`'s. Introduce features into the `pixi.toml` that can describe parts of environments.
As an environment goes beyond just `dependencies` the `features` should be described including the following fields:

Expand All @@ -40,7 +37,6 @@ As an environment goes beyond just `dependencies` the `features` should be descr
- `target`: All the above features but also separated by targets.
- `tasks`: Feature specific tasks, tasks in one environment are selected as default tasks for the environment.


```toml title="Default features"
[dependencies] # short for [feature.default.dependencies]
python = "*"
Expand Down Expand Up @@ -94,8 +90,8 @@ The environment definition should contain the following fields:

- `features: Vec<Feature>`: The features that are included in the environment set, which is also the default field in the environments.
- `solve-group: String`: The solve group is used to group environments together at the solve stage.
This is useful for environments that need to have the same dependencies but might extend them with additional dependencies.
For instance when testing a production environment with additional test dependencies.
This is useful for environments that need to have the same dependencies but might extend them with additional dependencies.
For instance when testing a production environment with additional test dependencies.

```toml title="Creating environments from features"
[environments]
Expand Down Expand Up @@ -134,8 +130,10 @@ lint = ["lint"]
```

### Lockfile Structure

Within the `pixi.lock` file, a package may now include an additional `environments` field, specifying the environment to which it belongs.
To avoid duplication the packages `environments` field may contain multiple environments so the lockfile is of minimal size.

```yaml
- platform: linux-64
name: pre-commit
Expand All @@ -159,8 +157,8 @@ To avoid duplication the packages `environments` field may contain multiple envi
...:
```


### User Interface Environment Activation

Users can manually activate the desired environment via command line or configuration.
This approach guarantees a conflict-free environment by allowing only one feature set to be active at a time.
For the user the cli would look like this:
Expand All @@ -181,6 +179,7 @@ pixi shell -e cuda
pixi shell --environment cuda
# Starts a shell in the `cuda` environment
```

```shell title="Running any command in an environment"
pixi run -e test any_command
# Runs any_command in the `test` environment which doesn't require to be predefined as a task.
Expand All @@ -200,10 +199,11 @@ pixi run test
- GitHub project: [#10](https://github.com/orgs/prefix-dev/projects/10)

## Real world example use cases

??? tip "Polarify test setup"
In `polarify` they want to test multiple versions combined with multiple versions of polars.
This is currently done by using a matrix in GitHub actions.
This can be replaced by using multiple environments.
In `polarify` they want to test multiple versions combined with multiple versions of polars.
This is currently done by using a matrix in GitHub actions.
This can be replaced by using multiple environments.

```toml title="pixi.toml"
[project]
Expand Down Expand Up @@ -289,10 +289,10 @@ pixi run test
```

??? tip "Test vs Production example"
This is an example of a project that has a `test` feature and `prod` environment.
The `prod` environment is a production environment that contains the run dependencies.
The `test` feature is a set of dependencies and tasks that we want to put on top of the previously solved `prod` environment.
This is a common use case where we want to test the production environment with additional dependencies.
This is an example of a project that has a `test` feature and `prod` environment.
The `prod` environment is a production environment that contains the run dependencies.
The `test` feature is a set of dependencies and tasks that we want to put on top of the previously solved `prod` environment.
This is a common use case where we want to test the production environment with additional dependencies.

```toml title="pixi.toml"
[project]
Expand Down Expand Up @@ -348,15 +348,15 @@ pixi run test
```

??? tip "Multiple machines from one project"
This is an example for an ML project that should be executable on a machine that supports `cuda` and `mlx`. It should also be executable on machines that don't support `cuda` or `mlx`, we use the `cpu` feature for this.
```toml title="pixi.toml"
[project]
name = "my-ml-project"
description = "A project that does ML stuff"
authors = ["Your Name <your.name@gmail.com>"]
channels = ["conda-forge", "pytorch"]
# All platforms that are supported by the project as the features will take the intersection of the platforms defined there.
platforms = ["win-64", "linux-64", "osx-64", "osx-arm64"]
This is an example for an ML project that should be executable on a machine that supports `cuda` and `mlx`. It should also be executable on machines that don't support `cuda` or `mlx`, we use the `cpu` feature for this.

````toml title="pixi.toml"
[project]
name = "my-ml-project"
description = "A project that does ML stuff"
authors = ["Your Name <your.name@gmail.com>"]
channels = ["conda-forge", "pytorch"] # All platforms that are supported by the project as the features will take the intersection of the platforms defined there.
platforms = ["win-64", "linux-64", "osx-64", "osx-arm64"]

[tasks]
train-model = "python train.py"
Expand Down Expand Up @@ -416,3 +416,4 @@ pixi run test
```shell title="Running the project on a machine without cuda or mlx"
pixi run train-model
```
````
52 changes: 27 additions & 25 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,36 +95,38 @@ extra:
version:
provider: mike


nav:
- Getting Started : index.md
- Basic Usage: basic_usage.md
- Reference (CLI): cli.md
- Configuration: configuration.md
- Environment: environment.md
- Vision: vision.md
- IDE Integration:
- PyCharm: ide_integration/pycharm.md
- In-depth:
- Authentication: advanced/authentication.md
- Tasks: advanced/advanced_tasks.md
- Multi Platform: advanced/multi_platform_configuration.md
- Info Command: advanced/explain_info_command.md
- Channel Logic: advanced/channel_priority.md
- GitHub Actions: advanced/github_actions.md
- Global Configuration: advanced/global_configuration.md
- Pyproject.toml: advanced/pyproject_toml.md
- Getting Started:
- Installation: index.md
- Basic Usage: basic_usage.md
- IDE Integration:
- PyCharm: ide_integration/pycharm.md
- Tutorials:
- ROS 2: tutorials/ros2.md
- Examples:
- C++/Cmake: examples/cpp-sdl.md
- OpenCV: examples/opencv.md
- ROS 2: examples/ros2-nav2.md
- Tutorials:
- ROS 2: tutorials/ros2.md
- Design Proposals:
- Multi Env: design_proposals/multi_environment_proposal.md
- Community: Community.md
- FAQ: FAQ.md

- Project configuration:
- Pixi.toml: configuration.md
ruben-arts marked this conversation as resolved.
Show resolved Hide resolved
- Pyproject.toml: advanced/pyproject_toml.md
- Features:
- Environments: environment.md
ruben-arts marked this conversation as resolved.
Show resolved Hide resolved
- Tasks: advanced/advanced_tasks.md
- Multi Platform: advanced/multi_platform_configuration.md
- Multi Environment: advanced/multi_environment.md
- Advanced:
- Authentication: advanced/authentication.md
- Info Command: advanced/explain_info_command.md
- Channel Logic: advanced/channel_priority.md
- GitHub Actions: advanced/github_actions.md
- Global Configuration: advanced/global_configuration.md
- Reference:
ruben-arts marked this conversation as resolved.
Show resolved Hide resolved
- CLI: cli.md
- Misc:
- Pixi vision: vision.md
- Community: Community.md
- FAQ: FAQ.md

plugins:
- search
Expand Down
Loading