Provide a 3-byte type with GIGO convertibility to/from char in zerovec

[hsivonen]

Please provide a type in zerovec that has these properties:

• Stored as 3 bytes.
• Converts from char.
• Converts to char such that if the value is outside the scalar value range, panics if debug assertions are enabled and returns U+FFFD if debug assertions are not enabled.

Use case: Storing non-BMP normalization expansions without wasting a byte per scalar value.

[hsivonen]

CC @Manishearth

[Manishearth]

This feels rather specific to the use case, I don't know if this belongs in zerovec but it can be designed pretty easily outside of zerovec, you just have to implement AsULE and ULE appropriately. Happy to provide guidance on that.

In general i intend zerovec to be a generally useful crate, not one specific for i18n, and as such I view APIs under that lens when considering them for inclusion.

[sffc]

This seems like a reasonable addition. I would say that this should be on the general type CharULE.

This can be done in two phases:

1. Change CharULE to from 4 bytes to 3 bytes
2. Change the validation behavior to use U+FFFD

We should do (1) in the short term, before 1.0, since it affects data file stability. This should be a fairly easy change to make. We can do (2) later, since it only affects runtime behavior.

[Manishearth]

So I'm very happy to have (1), I'm not as sure if we should have U+FFFD validation behavior. It seems a bit strange to me and rather specific, given that zerovec tends to avoid GIGO overall.

[sffc]

I split (1) into a separate issue in the 1.0 Polish milestone: #1970

We can keep this issue open for a future milestone to further discuss (2).

[Manishearth]

sounds good!

[hsivonen]

Thanks. 3 bytes per char is more important for my use case than whether the value space is checked on access (and mapped to U+FFFD) or checked at deserialization time (and fails deserialization) assuming that the deserialization check happens at most once over the lifetime of the process and doesn't happen on every normalizer or collator instantiation.

[hsivonen]

(I would also take a U24 that would just be a 3-byte integer.)

[Manishearth]

happens at most once over the lifetime of the process and doesn't happen on every normalizer or collator instantiation

(as we've mentioned before this is entirely dependent on the caching strategy/etc used but we do plan to support the ability to do this)

[Manishearth]

the u24 can be done without any unsafe by using RawBytesULE<3> and a U24 type that contains that type as its AsULE, with convenient getters

[hsivonen]

I implemented U24 for my immediate use case in #1967, but afterwards it could be hoisted into some place more generic.

[robertbastian]

I think it'd be confusing for CharULE to not work for all chars, which is what I understand would happen with @sffc's suggestion.

Wouldn't it be cleaner to define another type like BmpCharULE? This could live outside the zerovec crate so we could do all the GIGO we want.

[hsivonen]

I think it'd be confusing for CharULE to not work for all chars, which is what I understand would happen with @sffc's suggestion.

3 bytes in enough for any char.

Wouldn't it be cleaner to define another type like BmpCharULE? This could live outside the zerovec crate so we could do all the GIGO we want.

It would make sense to have a BmpChar as well. In particular, it would be extremely weird if the supplementary characters in normalizer data were validated at deserialization time but the BMP characters weren't.

[robertbastian]

Oh right it doesn't use UTF-8 internally. However, doesn't making CharULE a different size to char break things like storing &[T::ULE] and then interpreting that as &[T]?

[sffc]

Oh right it doesn't use UTF-8 internally. However, doesn't making CharULE a different size to char break things like storing &[T::ULE] and then interpreting that as &[T]?

This is not something that ZeroVec supports for any type (except implicitly by those where T::ULE == T).

ZeroVec has from_slice_or_alloc to map &[T] to &[T::ULE] based on EqULE, which will no longer be supported for char, but this is a methodd I want to phase out anyway: #1935

[sffc]

Discussion with @robertbastian @Manishearth @sffc:

• char should remain a deserialization-time validated type
• We can/should introduce UnvalidatedChar([u8; 3]), whose ULE is RawBytesULE<3>, which has functions to convert to char with various types of behavior
• No consensus on where this type should live; maybe it should live next to UnvalidatedStr, but we're not sure that's in the best place either
• Decision: Implement it next to UnvalidatedStr, but re-evaluate the location of both types before Utils 1.0