Integration with errors.haskell.org?

[david-christiansen]

errors.haskell.org presently documents error messages and warnings from GHC, starting with version 9.6.1 once it's released. The design, however, is intended to accommodate any Haskell tooling whatsoever. Would Stack be interested in integrating with the site?

What it entails:

• Each error or warning is assigned a unique code of the form [S-12345], where "S" is a namespace (we can talk about what it would be) and "12345" is a number that's unique in that namespace
• These codes are kept consistent over time, even if the text is rephrased.

As I see it, there'd be the following benefits:

• Users have an easier time contributing documentation for how to work around various situations that may arise, so more docs may exist
• Error messages gain a notion of identity that allows them to be rephrased without breaking all docs
• Interactive tools that invoke Stack and GHC can link to one consistent, predictable location if an error message occurs, and new users don't have to figure out which tool is the source of the error

The costs would likely be:

• A one-time cost to put in the codes and update all tests that refer to the string forms of errors
• An small but real ongoing burden to think about the relationship between messages in older and newer versions to keep the semantics of the codes consistent over time
• Some user-written documentation will be in a location other than the manual

What do you think? Is this interesting?

[mpilgrem]

I would be happy to support. There are only 55 occassions of logError in the Stack code base and quite a number of those are for multi-line error messages, so it would not take long to implement. One would probably also want to include Pantry, but it only has two occassions of logError itself.

I would, however, like to keep the code short. Could Stack 'claim' the 'S' namespace and Pantry the 'P' namespace?

[david-christiansen]

I think S for Stack makes tons of sense. Does it make sense to give Pantry its own namespace, though? The way I see it, the namespace is scoped to the tool, and from the perspective of a user, it shouldn't matter whether an error arises from the executable itself or from a support library that's developed in tandem with it.

WRT assigning the numbers, the GHC technique was to uniformly sample them from the space of allowed errors, which reduces the likelihood of conflicts in long-running branches, makes different codes very visually distinct, and doesn't encode any information about the message into the number, which could lead to issues in the future if a sub-namespace runs out of space. Of course, tools that use the error index are free to assign the codes however they want, and the HF is just administering the tool namespaces.

[mpilgrem]

It probably does not make sense for Pantry to have its own namespace. I guess like any tool that has dependencies, and the dependency throws an error that the tool does not itself deal with, some types of errors arising from the use of Stack will not have codes. EDIT: As Stack is built on top of Cabal (the library), it would be good for Cabal to participate.

[david-christiansen]

I opened a very similar issue with Cabal: haskell/cabal#8543

It seems plausible to me that really doing this right would involve having more structured representations of errors throughout the ecosystem, so that e.g. an exception handler in an executable that formats errors for display could assign the right code. This would be a nice side effect, but the cost could be too high at the moment. I think a best-effort approach is a good start to making things more useful.

[hasufell]

It seems plausible to me that really doing this right would involve having more structured representations of errors throughout the ecosystem,

This is hard. One way I tried in GHCup is to use open sum types (haskus Excepts by @hsyl20). But they quickly get out of hand and you end up building a hierarchy of open sum types (e.g. rewrapping Cabal error type in your own error type).

A very unergonomic example of this is: https://gitlab.haskell.org/haskell/ghcup-hs/-/blob/master/lib/GHCup/Errors.hs#L363

The problem with open sum types are: they're quite unergonomic. I know someone who wrote his thesis about this related topic of error handling in Haskell, maybe he could chime in.

I believe a convention will not be enough. It will have to be a library that simply leads programs to having the properties we want.

[mpilgrem]

Thinking about this further overnight, I realise that I am not across the population of errors that Stack itself reports and their nature. I am planning to document them in this repository, and then take stock.

[mpilgrem]

I've documented about 240 things that look to me like 'error messages' in Stack's code, at docs/maintainers/stack_errors.md. They include: data constructors of several types that are instances of Exception, use of throwString (StringException), use of error, other uses of throwIO or throwM (with types not defined by Stack), use of exitWith (and derivatives), and uses of logError where Stack does not terminate.

My working definition of an 'error message' was (a) something described as an 'error' and/or (b) something that provides the user with information and causes Stack to terminate without having completed what the user likely hoped for.

I am wondering if some tidying up (logical and editorial) may be required before unique codes are assigned to things.

[david-christiansen]

@hasufell I don't see why open sums are necessary here - doesn't a bog-standard datatype, like they have for GHC's structured errors, do the trick? In a big project like GHC there's four of them, and a type class that describes how to do things like dump them to stderr.

Open sums for exceptions are clearly compelling - there's a reason why exceptions are the only open sum in ML, for instance - but for purposes of error message processing an ordinary sum type seems reasonable enough to me.

[david-christiansen]

I am wondering if some tidying up (logical and editorial) may be required before unique codes are assigned to things.

That might be worthwhile - I know that @goldfirere spent a bit of time working out the exact assignment of codes to GHC errors, because sometimes further arguments to a sum type actually led to logically different errors.

It's also perfectly fine to do error index integration incrementally. If the 30 most common or most important errors are coded and documented, that's already a ton of value, even if many of them remain as strings for some time.

[hasufell]

doesn't a bog-standard datatype, like they have for GHC's structured errors, do the trick

If it's within your own codebase boundaries and everything else is wrapped and rethrown, certainly.

But if you want to observe and pass through errors from, say, Cabal the library in stack, it gets somewhat hairy.

I'm just brainstorming. Nothing of this is a blocker for implementing the proposed.

[mpilgrem]

Stack mostly dealt with its own exceptions by a series of module-specific (closed) sum types that were instances of Exception. I've now tidied up the remainder by converting them also to data constructors of those, or similar, types. There remain a handful of things that are reported by Stack as 'errors' but which are not thrown, and caught in Main.main, like the others.

In some cases, Stack would report a lengthy 'pretty' error and then throw a short 'black and white' exception. It seemed to me that there was a need for a PrettyException e, where the whole of the 'pretty' error message was part and parcel of the exception itself. I've implemented that in Stack but raised a pull request commercialhaskell/rio-prettyprint#12 to push it down into Stack's, and Pantry's, pretty logging dependency. Some (perhaps all, for consistency) of Stack's 'black and white' exceptions could be made 'pretty'.

I've realised, belatedly, that Pantry does have its own PantryException sum type - https://hackage.haskell.org/package/pantry-0.7.0/docs/Pantry.html#g:4, with about 40 data constructors.

I think the next thing I need to do is assign error numbers to Stack's exceptions. Following GHC's example, I plan to do that as follows: sample randomly from S-1000 to S-9999, without repetition. I can't imagine Stack could ever need more than four digits to uniquely identify an error. My thinking is that Pantry could make use of S-100 to S-999, to avoid the need to co-ordinate between Stack and Pantry.

[mpilgrem]

All errors that Stack itself generates now include a unique "Error: [S-nnnn]" in the output.

[david-christiansen]

This is really great! I've set up haskellfoundation/error-message-index#358 to track progress on the other side of getting this integrated.

Do you know what version number the first Stack release that includes these will have?