Use separate dependency IDs for pre-compiled and source dependencies

[robdimsdale]

Describe the Enhancement

Currently we use a single dependency ID for both source and pre-compiled dependencies. This leads to issues with selecting dependencies (e.g. paketo-buildpacks/packit#403) and leads to suboptimal output for users - who don't necessarily know if they are getting a pre-compiled binary or not.

We should explore using separate identifiers for the source and pre-compiled dependencies. This would likely look like the Build phase resolving both dependencies and prioritizing the pre-compiled dependency if one exists.

This would also easily allow us to improve the logging, as it will be more obvious to users which dependencies are getting resolved and ultimately selected.

[joshuatcasey]

What would happen if we put source dependencies in another buildpack entirely? (Just as a mental thought exercise).

[robdimsdale]

What would happen if we put source dependencies in another buildpack entirely? (Just as a mental thought exercise).

I suppose we could do that. We would have the source buildpack non-optional in every order group and the pre-compiled buildpack optional in every order-group. But to achieve that, we'd have to find a way to disable the build phase of the source buildpack when the pre-compiled buildpack participates in the build plan. Or we would have to duplicate every existing order group to support either pre-compiled or source compilation. Both of those sound problematic.

Setting aside the order group issue, splitting the buildpacks does have the advantage of simplifying the pre-compiled buildpack. However, it just moves the complexity elsewhere. Any common logic would have to be duplicated or extracted to a shared library. I'm not convinced it's worth the tradeoff.

Finally, as an aside, splitting out the dependencies into a separate buildpack or pre-provided layer is an idea that some folks have started to think about. It's part of a bigger exploration about how we could separate dependencies from the buildpacks that install them. The latest thinking on this topic is in this google doc.

[robdimsdale]

I'm going to move this to the cpython repo as it's the only repo that this affects.

[sophiewigmore]

I'm also wondering what this would look like for the dependencies automation. Everything went through many iterations and I know we considered this use case * at some point * but I'm not sure if the workflow is fully equipped to handle it. I'm down to do a quick investigation into this and make sure things are functional

[sophiewigmore]

Following up - are you blocked at all on that investigation @robdimsdale ?

[robdimsdale]

No, I don't think we're blocked on the investigation.

My initial foray into this suggests that it might be a fair bit of work to support multiple dependencies with different IDs.

Firstly, the cpython dependency retrieval code is hard-coded to handle exactly one dependency with the ID python. That is fairly easy to change to support two hard-coded dependencies with different IDs.

Following the thread a bit further, it seems like the workflow might not handle multiple dependencies well. I haven't dug into it, but I suspect the matrix job definition is where we might encounter issues.

As part of this work, I'll explore any necessary changes to the workflow to enable multiple dependencies with different IDs. I think the outcome is likely worth the effort.