Support for bytecode files and libraries

[mpickering]

This proposal describes extensions to Cabal and cabal-install to support building, installing, and managing persistent bytecode artifacts (.gbc files) and bytecode libraries alongside native code for faster development workflows.

• GHC #26298
• Cabal #11188
• Discourse post: RFC: Introduce a serialisable bytecode format and corresponding bytecode way

Rendered

[zlonast]

Rendered 👀

[mpickering]

Rendered 👀

Thanks, I added the link to the main post.

[chrisdone]

Small question: is the byte code inherently platform or architecture specific? Or is it portable? Just curious.

[geekosaur]

I'm curious about that as well: while technically bytecode needn't be architecture-specific, word size and endianness can still affect it.

(To be clear, this isn't a Cabal question or directly related to the proposal; it might affect uptake, but I doubt it since it would be the only piece of a registered package that would be portable anyway.)

[geekosaur]

Consider this conditional, since I'll want to also review the GHC ticket/MR. But taken by itself, it shouldn't be much more than a cut-and-paste of the vanilla build way. (Which does make me wonder if we have a sensible architecture for ways in place, but that's not really part of this proposal as such.)

[mpickering]

Small question: is the byte code inherently platform or architecture specific? Or is it portable? Just curious.

Inherently, I think the bytecode part is quite portable but there are a few things to consider which would make it impractical now.

• Platform dependent CPP
• Having the same libraries available (which expose exactly the same symbols) on each platform
• Endianess/wordsize of the generated bytecode
• Foreign C calls, and calls to other object code

I think @TerrorJack and @alt-romes could add more

[angerman]

Interesting proposal! I've also been concerned about it being platform specific, however I believe this is completely irrelevant, because cabal only installs these local, compiler specific (and thus platform specific) store? Am I mistaken?

I am however reminded about the Fat Interface Files discussion we had a few years back that Josh was working on. If gbc was part of .hi files, and .hi files were containers, we wouldn't need a change to cabal at all, right? And all of this could live exclusively in GHC?

[geekosaur]

I don't think this would fit in fat interface files (which exist now).

And yes, these would be in package dbs, which are already host-specific. As, come to think of it, are the .hi files.

[geekosaur]

As far as implementation, I think haskell/cabal#9900 could be used as a template?

[angerman]

I don't think this would fit in fat interface files (which exist now).

And yes, these would be in package dbs, which are already host-specific. As, come to think of it, are the .hi files.

Why not? Sounds to me like the bytecode is technically metadata for the compiler (same as the interface files), and if it was in the interface files, the whole change would be transparent to cabal and self-contained in ghc?

[geekosaur]

You'll have to make a ghc proposal for that, though. The current plan is to treat it as a new object format and compiler way, and that will require cabal (and stack) support.

[angerman]

You'll have to make a ghc proposal for that, though. The current plan is to treat it as a new object format and compiler way, and that will require cabal (and stack) support.

Yes, I'm questioning the premise here.

I don't see why you do. The hi file format is internal to ghc; I do not believe there are any guarantees around the .hi file format. It's an opaque blob to tooling.

I don't think there are any interoperability guarantees for it. Would MicroHS read ghc .hi files?

Again, ghc bytecode is internal to GHC, and if it was part of the hi files, it would be transparent to tooling, and automatically supported by cabal and stack (and any other tool dealing with ghc build products, nix, bazel, buck, ...)

And again, if bytecode is considered internal to GHC (and I don't think we have a haskell bytecode spec?), then I'm not convinced it would need a ghc proposal.

[geekosaur]

The hi file format is internal to ghc; I do not believe there are any guarantees around the .hi file format. It's an opaque blob to tooling.

stack reads .hi files, and in fact mpilgrem has asked the ghc devs to rearrange the .hi file format before GHC 9.14.1 is released because something it needs now lives at the end of the file.

It also seems to me that adding bytecode to it (a) complicates compilation (does that mean bytecode is added even during normal compiles?) and (b) slows compilation in general because .hi files need to be read for imported modules. But, again, this should be raised with the GHC devs, not here.

[angerman]

I am raising this on the ghc issue tracker as well. I do raise it here because if it's part of the .hi file. This proposal, and any subsequent change to cabal or stack could be cut short.

stack reads .hi files, and in fact mpilgrem has asked the ghc devs to rearrange the .hi file format before GHC 9.14.1 is released because something it needs now lives at the end of the file.

Maybe this is a great opportunity to spec a .hi container format if we want to make it publicly readable. Seekable containers are a fairly common pattern?

[geekosaur]

I have a feeling that if the ghc proposal is advanced enough that they're proposing cabal changes to support it, it's too far advanced to be redirected at this point.

And I think mpilgrem proposed a container format for .hi files as well.

[angerman]

I have a feeling that if the ghc proposal is advanced enough that they're proposing cabal changes to support it, it's too far advanced to be redirected at this point.

I don't see a ghc proposal. I only see a ghc issue tracking it? Is there a ghc-proposal for this I'm not aware of?

[geekosaur]

Hm. I would have expected there to be, but indeed there seems to be only the tracking issue and Discourse RFC.

[ffaf1]

At today’s Cabal-dev meeting there were no objections about this.

This proposal is accepted.

[angerman]

At today’s Cabal-dev meeting there were no objections about this.

This proposal is accepted.

So these proposals will just be voted on, without discussions on the proposal being addressed?

Is there a recording of said cabal-dev meeting?

PS: I don't mind if the cabal-devs, proposer, or someone else deems raised concerns irrelevant, misguided, or something else entirely, but just outright ignoring them has me questioning the point of this repository and process.

[ffaf1]

Hello Moritz,

Cabal-dev meeting notes are here.

As per Cabal Proposal Process the Cabal-dev meeting does not debate on the technical merit of the proposal, but only if rough consensus has been reached in the proposal PR.

Evaluating both this thread and the one on Discourse, we felt the proposal has rough consensus.

Cabal-proposal process is still new, any feedback is welcome to improve the process.
Rereading the thread I understand your complaints. I will bring those to the next Cabal-dev meeting.

[ulysses4ever]

I found Brandon's assignment above convincing, but in the hindsight we should have waited for @angerman to react. Indeed, this is a new process, and we're still learning.

[angerman]

I don't consider any of my concerns addressed. My fundamental question remains: why does this need to be in cabal and can not be self contained in ghc. The proposal doesn't address this, and the discussion doesn't address this.

I've had some discussions with others around this, and there are some reasons for why this path might have been chosen, yet it's not reflected anywhere.

If this proposal process is supposed to be meaningful we need to have raised questions addressed and this could perfectly fine be the cabal devs saying they consider these questions to be irrelevant, made in bad faith, unproductive, or something else entirely; not addressing or acknowledging them in any form leaves me wondering what this process is for.

Let's hope we can all learn from this!

I would still very much like to see a rational for why this logic needs to live in cabal, and can't be contained within GHC itself. Any added complexity is a liability and my understanding is that this process is precisely here to help with this. Maybe I'm wrong.

[geekosaur]

I will repeat that that is not a Cabal question, it is a GHC question. The GHC team made its decision and launched a Cabal proposal based on it; it is not our place to question the GHC team's decision, only to consider its impact on Cabal.

[hasufell]

The GHC team made its decision 

That's not a reason to accept the proposal though. Otherwise it seems GHC can dictate whatever they prefer to push down into cabal.

I feel this proposal process lacks the very fundamentals of a process: discussing the technical merits.

[mpickering]

The discussion about the separate files or not is continuing over here: https://gitlab.haskell.org/ghc/ghc/-/issues/26298#note_637273

[geekosaur]

I will further add that I think GHC needs a process for this kind of thing. Wider dissemination should be part of it, and the process will give stakeholders a single place to monitor. (Why, for example, would arbitrary GHC users think to look in Cabal's proposals for information about a new GHC feature?)

Other things that could benefit:

• TH
• the ghc-api stability project
• now that base is decoupled from GHC, perhaps it should be handed over completely to the CLC and this process be used to negotiate GHC's support for base features

[mpilgrem]

@angerman, thanks for looping me in.

There is a lot to read up on, but I think that Stack is likely downstream of choices made by Cabal (the library).

As an aside on Stack and GHC's *.hi files (referenced in @geekosaur comments above): originally, I thought that parsing the new GHC 9.14 format was going to be a pain. However, once I understood it better, I realised the *.hi files contained information that enabled Stack to skip the blocks of complex data it was not interested in without having to parse it. So, ultimately, reorganization was not needed to facilitate Stack's needs.

[angerman]

There is a lot to read up on, but I think that Stack is likely downstream of choices made by Cabal (the library).

Yes, that's pretty much my understanding as well. I just wonder if it wouldn't be better if this wasn't cabal-proposals, but these kinds of proposal would be rather in haskell-ecosystem-proposals, where the relevant stakeholders, including stack, would be asked for input.

Given the general consensus(?) that this repo is for

If cabal-proposals is for:

• cabal only changes, I can see it make sense.

This specific proposal seems to be straddling the lines between GHC and Cabal. I think my biggest concern here is that, it's asking for downstream changes due to an upstream change, but it's unclear what the input on this proposal is supposed to be here other than "Ok 🤷‍♂️".

I can see some use out of this repo for conceptual/structural changes to cabal itself, where proposals of restructuring and similar of cabal is concerned. For items like the one under discussion, it seems we'd rather want a ghc+cabal+... proposal to get agreement on altogether (if we want to go down the proposal route).

[angerman]

After writing the last comment, I feel more confused than before.

We seem to have two situations:

1. We want to discuss an idea (without a concrete implementation, or partial implementation).
2. We want to discuss a concrete change. (E.g. we have a concrete implementation with Pull Requests).

For (1) I can understand the proposal process can help build consensus. However, even then it's all fairly hypothetical at that point, there might still be a lot of unknown-unknowns surfacing during the implementation, which could end up contradicting assumptions? The benefit would be that general consensus of direction is reached, and buy-in from the relevant stakeholders?
For (2) I'm not sure what the proposal process adds over discussing the concrete/actual implementation in the PR? Say I have a rough idea how something could work, but even I am not sure if and how, so I end up trying out a lot of things, probably throwing away many failed attempts in the process, and finally end up with something I am content with and happy to share/discuss? The only slight benefit I see the proposal process being able to offer here is maybe give input on some dead-end routes? Opening a proposal after opening the concrete PR seems like duplicating work and fraying the discussion?

[hasufell]

I've been in open source for nearly two decades and have worked on projects bigger than Cabal and GHC, so I'm familiar with different types of processes.

But I do not understand this process.

[ffaf1]

I understand the topics are intertwined, but we need to have a thread/PR about each problem:

1. Support for bytecode files and libraries in Cabal;
2. how GHC development impacts other tools in the ecosystem;
3. usefulness and process of cabal-proposals.

This thread is over fifty-messages long, and indeed there is a lot to read up on.

If nobody objects, I can split the latter in a new issue inside this repo, but I wouldn’t know where 2. should be (GHC gitlab)?

[mpickering]

Thanks for raising your concerns, @hasufell. I think it can be summarised that if you want to make a significant change to cabal then make a proposal to discuss the change. I can answer any specific questions if you have them.

@angerman

The question about "haskell-ecosystem" proposals is a much bigger question than "cabal-proposals" can answer. It seems to me that the hardest part of establishing that such a process would have legitimacy to establish direction with the large number of projects in the community. For instance, I would imagine that projects which may be interested would be GHC, cabal, stack, GHCup, haskell.nix, HLS, Haskell Nixpkgs infrastructure, hackage etc
All these projects have different developers, decision making procedures and processes.
Centralising decisions like this would be a significant departure from how the community is structured.

The current community model is decentralised and built upon the principle of trust and cooperation.
GHC developers make decisions about GHC. Cabal developers make decisions about Cabal. Cabal developers trust GHC developers to take their viewpoints into account when making decisions and GHC developers trust Cabal developers to integrate new features into Cabal and cabal-install to expose them to users. If that level of trust has broken down then we should talk about that openly and think about ways to rebuild it.

@angerman

In response to your (1) vs (2) comment, it seems that we can agree about the utility of (1).

For (2), I see your point that much of the discussion will naturally happen on the PR. Still, I think the proposal process does add something here.

• It ensures visibility across projects and allows non-engaged experts and stakeholders to offer their opinion on a human-readable specification.
• Provides a single place to record the rationale and feedback (which can be linked from PRs)
• It makes sure that decisions don’t get buried in an implementation thread.

In other words, it’s more about capturing agreement and documenting trade-offs at the "design intent" level.

I think an example of this process is in the implementation of bytecode objects. I have already prepared the code for that patch, but we are discussing and establishing the specification separately on the issue. I find that process really useful.

Thanks for bringing up these questions, the process is still new and it's useful to establish what people should expect when writing a proposal.

[TeofilC]

We have a variety of proposal processes in the Haskell ecosystem.

• GHC proposals are decided by the GHC steering committee
• HF ecosystem/technical proposals are decided by the HF
• Now, cabal proposals are decided by the cabal devs

I think the variety of proposal processes reflects that we have a variety of self governing projects (Conway's law).

We all talk to each other but ultimately decisions are made by these individual groups. For instance, I think it's good for stack devs to comment on cabal and GHC changes. It is not clear to me why we need a separate cross cutting decision process to allow that to happen.

(This overlaps a bit with Matt's points. I wrote this before seeing that but couldn't send it due to lack of internet)

[hasufell]

It ensures visibility across projects and allows non-engaged experts and stakeholders to offer their opinion on a human-readable specification

I don't think it achieves that. You're expecting people to monitor an additional repository, read the proposals and know when it matters to them and when their input is required. Anyone doing that is already an engaged expert.

To me, this process looks more like a way of creating documentation. I don't consider it a decision making process. And that's why it's super confusing...

• we don't know who those experts are
• we don't know who makes the decision... maybe it's the cabal devs... or maybe not, because they're just trying to understand if those "experts" reached consensus

And this thread is proof of that. Matt created a proposal, some people commented, apparently were not deemed "experts" and their comments ignored and a decision reached anyway.

This has the risk of exacerbating bias. Because both the "experts" and the "decision makers" are undefined.

[mpickering]

I think we’ve agreed in this thread, @hasufell, that a mistake was made in accepting this proposal prematurely. The assessment of consensus failed, since it wasn’t evident to everyone in the discussion that all concerns had been addressed. That’s an opportunity to learn from and improve the process for future proposals.

People who are interested in Cabal development can follow this repo and offer comments. Where needed, opinion can also be solicited explicitly by mentioning individuals or referring them to the thread. @angerman has done a great job of that in this thread by involving @mpilgrem. There is a section in the proposal to identify who the stakeholders are, which provides the opportunity to reflect on who will be affected by a change. Ultimately, the process relies on trust that Cabal developers will make decisions with the broader ecosystem in mind.

At this stage, my understanding is that Cabal developers don’t want a more formal process with defined "experts" and "decision makers" - the added bureaucracy would outweigh the benefits. The aim here is to keep things lightweight, to gather consensus and document rationale without slowing decisions. The alternative I see is to return to the previous model, which often led to very delayed decisions (or no decisions at all).

[geekosaur]

Centralising decisions like this would be a significant departure from how the community is structured.

I don't see it as about centralization, but synchronization. If the stakeholders aren't on the same page, the ecosystem has a problem; a centralized venue that stakeholders can easily follow for cross-project issues would help everyone impacted by such changes. But instead of centralized decision making, it's where people would discuss changes and iron out differences.

[angerman]

I don't think I have anything more to contribute here. The relevant ticket for discussion seems to be ghc!26298.

The consensus seems to be that this repo is cabal-only proposals, and not ecosystem proposals.

In other words, it’s more about capturing agreement and documenting trade-offs at the "design intent" level.

This seems to be the primary misunderstanding on my part. This repository is about capturing agreement and documenting trade-offs. Not discussing the implementation. I will treat it as such going forward.

It would be great if this proposal would then also document the trade-offs as discussed in ghc!26298? Or is a reference from the proposal sufficient?

I've raised my concerns, and they have been largely deemed out of scope.

[geekosaur]

This repository is about capturing agreement and documenting trade-offs. Not discussing the implementation. I will treat it as such going forward.

That's essentially what we agreed on in the proposal for it. I'm not sure I agree fully, because I'm not sure it actually accomplishes much, but everyone else went that way so that's what was done.

[mpickering]

It’s been a few weeks since the last round of comments, and it seems we’re in agreement that this proposal’s scope is limited to adding first-class support in Cabal and cabal-install for producing bytecode.

I have responded to all the comments I believe were outstanding. In particular there were questions by Chris and Moritz about particular aspects of the design.

Given that consensus, I’d like to resubmit the proposal for consideration at the Cabal developers meeting tonight.

@ffaf1 @Mikolaj does that sound good to you?

[ffaf1]

Sorry for the delay. The point was not raised at the last cabal-dev meeting.
For sure this will make next meeting.

@angerman asked about writing down trade-offs. I wonder whether there are trade-offs that are cabal specific.
If so, it would be nice to have them pinned down.

[Mikolaj]

@mpickering: Today at the cabal devs chat the attendees agreed to accept this proposal of changes to the cabal codebase.

This is going to complicate cabal to some extent, but we haven't found a profound enough reasons to block the GHC changes that require the cabal changes in question nor do we want to offer alternatives to the presented design (at this stage at least).

Please let us know if the design of the GHC part of this work is modified in a way that impacts cabal. Please feel free to use this ticket for further discussion and clarification (regardless if it gets closed or not).

[geekosaur]

In particular, I can see reasons for the proposed implementation and that any commercial users asking for it would want this implementation.

[mpickering]

Thanks for considering the proposal and Moritz and Julian for your discussion on it!