proposal: spec: generics: type initialization with value parameter #67064
Comments
smyrman
added
LanguageChange
|
The general idea has merit. The specific syntax here doesn't seem great. We can't support func NewMatrix[T any](n, m int) [n][m]T {
return [n][m]T{}
}That would appear to require us to compute types at runtime. But having two separate type parameter lists doesn't seem great either. Also it's not clear that the use cases for this are strong enough to support the additional language complexity. |
|
A possible alternative syntax, could perhaps be prefix related. I.e. since we already support // using = for value...
func NewMatrix[T any, N, M =int]() [N][M]T {
return [N][M]T{}
} |
|
We could perhaps support generic parameters that are values. We wouldn't want to use func NewMatrix[T any, N, M const int]Are there concrete use cases for this? The enum example doesn't seem to bring clear value. The matrix example might in principle be useful for SIMD, but would it really help in practice? In particular, we don't want to add dependent types to Go. That is complexity beyond what we want in the language. -- for @golang/proposal-review |
|
The use-case that made me think of this, was indeed the enum case. Or to be more specific, it's the use-case of translating between different enum representations in different layers of the application, and to minimize boiler plate when doing so by bake the functionality into a library (and through this proposal, maybe a language feature). Let's do an illustrative example. Consider three representations for a data resource "Food" in three different layers (clean architecture style):
We will only look at the first two layers, as there is no fundamental difference between the DB layer and the API layer; the DB layer is just encoding the information into a different wire format then the API layer. Let's start with the Business logic layer. The goal of this layer is to create a type that is agnostic to the encoding used in either the API layer (could have multiple maintained versions for backwards compatibility), or database (could have separate enum set for legacy reasons and/or multiple implementations). package entities
type FoodType uint
const (
_ FoodType = iota
FoodTypeAubergine
FoodTypePasta
FoodTypeFish
)
type Food struct{
...
Type FoodType
}There is no text representation in this layer, as that's irrelevant to the business logic implementation. For each API version a separate Food struct is defined, focusing on encoding to e.g. JSON. There is also a separate foodType lookup defined, including alias support for parsing legacy values or alternate spellings. So in package api
foodTypes = enum.NewLookup(map[entities.FoodType][]string{
entities.FoodTypeAubergine: {"aubergine", "eggplant"},
entities.FoodTypePasta: {"pasta"},
entities.FoodTypeFish: {"fish"},
})
type Food struct{
...
Type textfield.Enum[foodTypes] // Implements TextMarshal / TextUnmarshal
}
func (ft *Food) FromEntity(e entities.FoodType) {
...
ft.FoodType.V = e.FoodType
}Assuming library code: package textfield
type Enum[L var enum.Lookup[T], T comparable] struct{
V T
}
func (e Enum[T,L]) MarshalText() ([]byte, error) {...}
func (e *Enum[T,L]) UnmarshalText(data []byte) error {...}Comparing to a solution using only existing language features and with library helpers for encode and decode, you wouldn't save more than ~7 lines of code per enum, per API version. But it does add some cognitive overload. Optimized example without new language features: package api
foodTypes = enum.NewLookup(map[FoodType][]string{ // ~ changed
entities.FoodTypeAubergine: {"aubergine", "eggplant"},
entities.FoodTypePasta: {"pasta"},
entities.FoodTypeFish: {"fish"},
})
type FoodType structPentities.FoodType // + added
func (ft FoodType) MarshalText() ([]byte, error) { // + added
return enum.MarshalText(foodTypes, ft) // + added
} // + added
func (ft *FoodType) UnmarshalText(data []byte) error { // + added
return enum.UnmarshalText(foodTypes, ft, data) // + added
} // + added
type Food struct{
...
Type FoodType // ~ changed
}
func (ft *Food) FromEntity(e entities.FoodType) {
...
ft.FoodType = FoodType(e.FoodType) // ~ changed
}So is it worth it? In real numbers and a real product (Clarify.io), I count at least 12 enum types in a single service in our backend code base. We are in the process of moving to clean architecture there. On enums alone, we could, with this language feature, save ~170 lines of boiler plate if we consider only one DB version and one API version. Considering other use-cases (not enum based), we can reduce another ~60 lines by removing a 3 line function from ~20 entity types. Clean architecture is heavy on boiler plate, so places where we can reduce it, seams worth exploring. |
|
Generic value parameters comes extremely useful when you want create a type with a static callback function. That is - currently you need to pass it to the type constructor and store It inside the struct. This is especially painful for something that accepts static functions as comparators. Generic value parameters will allow you to write some generic type with sorting on top of the slice type without wrapping it in the struct. |
|
Just as a side-note... I think that if we where to allow func init() {
lib.RegisterCallback(mykey, myCallback)
}
type MyType = lib.TypeWithCallback[myKey] |
|
I found another use-case that I want to highight, where a variant of this feature would be useful. Recently, I have been trying out the pattern of encapsulated fields similar to what's used in ardanlabs/service. A basic example of what an encapsulated field is, can be found here. The essential portions of the linked code is provided here: // Name represents a name in the system.
type Name struct {
name string
}
// String returns the value of the name.
func (n Name) String() string {
return n.name
}
var nameRegEx = regexp.MustCompile("^[a-zA-Z0-9' -]{3,20}$")
// ParseName parses the string value and returns a name if the value complies
// with the rules for a name.
func ParseName(value string) (Name, error) {
if !nameRegEx.MatchString(value) {
return Name{}, fmt.Errorf("invalid name %q", value)
}
return Name{value}, nil
}Reasons for using such a field in a (business layer) model, are similar to the reasons of using Here is an example model using encapsulated fields: type User struct {
ID xid.ID
Active bool // doesn't need encapsulating; already guaranteed valid
Name Name // encapsulated string
Email Email // encapsulated string
ProfilePicture ProfilePicture // encapsulated string
}How value parameterization could helpWith this type of programming, I end up with many similar fields with slight differences. While this is an expected trade-off, it also means that this could be a candidate for generics. Usually there are categories of field types that could be initialized with a variable to reduce boiler plate. For most cases this variable could be For the case above, we could picture: type PatternString[p const string] struct {
value string
}
func (n PatternString[p]) String() string {
return n.value
}
func ParsePatternString[p](value string) (PatternString[p], error) {
if !regex.MustCompile(p).MatchString(value) { // could be optimized, but this works.
return PatternString[p]{}, fmt.Errorf("value %q does not match pattern %q", value, pattern)
}
return PatternString[p]{value}, nil
}With initialization: const patternName = "^[a-zA-Z0-9' -]{3,20}$"
var _ = regexp.MustCompile(patternName) // ensure the pattern is valid.
type Name = PatternString[patternName]
func ParseName(name string) (Name, error) {
return ParsePatternString[patternName]
}I could now add type Email = PatternString[patternEmail]
func ParseEmail(name string) (Email, error) {
return ParsePatternString[patternEmail]
}type ProfilePicture = PatternString[patternProfilePicture]
func ParseProfilePicture(name string) (ProfilePicture, error) {
return ParsePatternString[patternProfilePicture]
}Another variant: // let the proto-type be private
type patternString[p const string] struct {
value string
}Use inheritance so that a filed can be extended: type ProfilePicture struct{
patternString[patternProfilePicture]
}
func ParseProfilePicture(name string) (ProfilePicture, error) {
return ProfilePicture{parsePatternString[patternProfilePicture]}
}
func (p ProfilePicture) URL() url.URL {...}
func (p ProfilePicture) PreviewURL() url.URL {...} |
|
Like for the enum and callback cases, the encapsulated field case above would ideally get passed a variable; in this case a However, adding type StringRegexp[re var *regexp.Regexp] struct{}Then: type StringPattern[pattern const string] struct{}would still go a long way. |
ianlancetaylor
changed the title
ianlancetaylor
added
LanguageChangeReview
|
In the original post, I see func (v Enum[L,T][lookup]) Value() (driver.Value, error) {What is |
|
OK, we see it now. The |
|
What we see here is func MustCreateLookup[T comparable](map[T]string) EnumLookup{
// implementation doesn't matter.
})
type MyType = sqlenum.Enum[uint](enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})This appears to be actually calling a function at compile time. That can't work. What if the function panics? Or behaves differently in different calls. There is too much flexibility in Go to permit calling functions at compile time. Or, perhaps it is being called at run time. But then why not just use an ordinary function? There is no obvious reason why a dynamically called function should be part of a type in some way. What advantage do we get from that? |
|
I'm not sure I really understand this proposal. Let me try to summarize it in my own words (a little bit stream-of-consciousness style, apologies) and tell me if I'm wrong. The proposal is to add a second list of (let's call them) "value parameters" to type declarations. Instead of interfaces/constraints, the arguments would have a type and would be expected to be instantiated with a value. If a generic type is instantiated with two different values, they would constitute different type. That implies (as the compiler needs to be able to type-check), that either a) the values used to instantiate this second parameter list have to be constants and hence the types of value parameters must have underlying type one of the predeclared non-interfaces. Or b) that we'd introduce some way for the compiler to do inference on the possible values that a value argument can take, which would be dependent types, which Ian excluded from consideration. So value parameters would always be constants. Now, what I'm struggling to understand is, what problems this solves and how. It seems the most interesting thing to talk about are the enum changes. The proposal text contains this: package enum
// Lookup provides a two-way lookup
// between values of type T and strings.
type Lookup[T comparable] interface{
Value(string) (value T, ok bool)
Name(T) (name string, ok bool)
}
func MustCreateLookup[T comparable](map[T]string) EnumLookup{
// implementation doesn't matter.
})Now, the comment says "implementation doesn't matter". Am I understanding correctly, that the implementation would be something like this? type EnumLookup[T comparable] struct {
names map[T]string
vals map[string]T
}
func (e EnumLookup[T]) Value(name string) (T, bool) {
v, ok := e.vals[name]
return v, ok
}
func (e EnumLookup[T]) Name(val T) (string, bool) {
s, ok := e.names[val]
return s, ok
}
// note the extra type parameter on `EnumLookup`, which the proposal text omits but seems necessary
func MustCreateLookup[T comparable](m map[T]string) EnumLookup[T] {
names := maps.Clone(m)
vals := make(map[string]T)
for val, name := range names {
if _, ok := vals[name]; ok {
panic(fmt.Errorf("duplicate name %q", name))
}
vals[name] = val
}
return EnumLookup[T]{names, vals}
}The proposal text then suggests a new library package // note: we are not actually allowed to make the underlying type T. We need to make it struct{v T}.
// But let's ignore that for now.
type Enum[L enum.Lookup[T], T comparable][lookup L] T
func (v Enum[L,T][lookup]) Value() (driver.Value, error)
func (v Enum[L,T][lookup]) Scan(src any) errorNow, I'm not sure how this is supposed to work. As I mentioned above, ISTM for this proposal to meaningfully work, value arguments must always be constants and value parameter types must be basic types. So, already, the implementation I suggested above doesn't work - it is based on structs and maps and those aren't constants. Thus, we need a different implementation for the Now, if we can make all of this work, I'd compare it to this: package enum
// as before (in the "no language change" section)package enumsql
func Scan[T any](e enum.Lookup[T], p *T, v any) error {
s, ok := v.(string)
if !ok {
return fmt.Errorf("unsupported type %T", v)
}
val, ok := e.Value(s)
if !ok {
return fmt.Errorf("invalid value %q", s)
}
*p = val
return nil
}
func Value[T any](e enum.Lookup[T], p *T) (driver.Value, error) {
s, ok := e.Name(*p)
if !ok {
return nil, fmt.Errorf("invalid value %v", *p)
}
return s, nil
}package main
var myTypeLookup = enum.NewEnumLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})
func (v MyType) Value() (driver.Value, error) {
return enumsql.Value(myTypeLookup, &v)
}
func (v *MyType) Scan(src any) error {
return enumsql.Scan(myTypeLookup, v, src)
}So, ISTM that what this proposal saves is having to implement two trivial methods, that just delegate to a helper. That's not nothing, but it also doesn't seem a lot. An alternative way to solve this with today's language is to instead make the type Value interface {
sql.Scanner
driver.Valuer
}
func Wrap[T any](e enum.Lookup[T], p *T) ValueThis has the cost of requiring an additional So I can see that there is some cost saved, with all these solutions. But ISTM the real utility here isn't so much to have "value parameters". The benefit does not really depend on |
I think there is mistake in the issue description, and the syntax should be with square brackets: type MyType = sqlenum.Enum[uint][enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
}]Where the first list of square bracket parameter with this syntax are type parameters, and the second list are variables. Which should be understood by the compiler as: var lookup = enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})
type MyType = sqlenum.Enum[uint][lookup]I would not expect the function to be run at compile-time, but at package initialization time (in this case). However, the return "variable" reference (type and address) must be resolved at compile time. Defining the variable inline is not essential to the proposal; it's just a way to make the variable scoped to the type declaration. I think the relevant cases to do type declarations with inlined variable references are:
On type equality, types are only equal if they are initialized with the exact same variable reference. Example of equal types: var lookup = enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})
// MyType1 and MyType2 are equal.
type MyType1 = sqlenum.Enum[uint][lookup]
type MyType2 = sqlenum.Enum[uint][lookup]Examples of unequal types: var lookup1 = enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})
var lookup2 = enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})
// MyType1 and MyType2 are NOT equal.
type MyType1 = sqlenum.Enum[uint][lookup1]
type MyType2 = sqlenum.Enum[uint][lookup2]This also means that the following two types are not equal: // MyType1 and MyType2 refer to different addresses and are _not_ equal.
type MyType1 = sqlenum.Enum[uint][enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})]
type MyType2 = sqlenum.Enum[uint][enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})]
The advantage I see, comes when declaring field types for use with models. When you define a model, you typically don't want to initialize it with information of how each field should be parsed and/or validated; instead you want the zero-value to be self-descriptive, and you want each model field to contain enough information about how it should be treated, without initialization. In other words, value parameters are trying to solve the same problem as struct tags, but for more cases. A struct-tag uses string-like syntax to tell libraries (that must rely on reflection) how to treat each field of the struct. This is additional (and constant) information that is embedded along with other information on the field, such as it's name and type. Value parameters aim to allow plain code (without reflection) to do exactly the same. By declaring a struct field as Use-cases within model field definitions could be:
For the record, I like the proposed syntax in this comment better. I am picturing a variant of that syntax that accepts type Enum[L enum.Lookup[T], T comparable, lookup var L] TBut what I like with syntax described by @ianlancetaylor , is that it allows controlling scope. E.g. start with As far as I can see, a |
|
@smyrman I think it would be useful to phrase your proposal in terms that are established in the spec or at least widely used in the community (and, of course, operatively defined where you introduce them newly, like "value parameter"). Note that Go doesn't have the notion of a "reference". And that is not nitpicky: It has pointers, but in general, there is no way to tell at compile time whether two pointers are equal. And not everything can be made into a pointer (that is, not every expression is addressable). Being precise about these things would naturally ask (and hopefully answer) some of the questions that have come up in this discussion. AIUI, you are suggesting in your last comment that
Did I understand that correctly? I'll note that under those semantics, there is no way to refer to
For the record, that is only the case if we restrict the possible expressions that can be used to instantiate a value parameter, roughly as I said above. General expressions can not be resolved at compile time. |
|
@Merovius, let me reply to you comments as well. Will start with the last one. 1 and 2 seams reasonable restrictions. We could add more restrictions as well, like only allowing const parameters to be declared in type aliases, certainly for a proto-type. In general, I think a requirement should be that a proto-type can generate code that will compile and run today. Anything that isn't consistent with that, should be considered out of scope.
Correct. You would have to use type aliases to be able to refer to concrete initialized types outside of the package (unless you expose your package variables). If the expression is a regex for instance, that makes a lot of sense; you need the type alias when dealing with such values. Or rely on the interface that is implemented by these types. Finally, let me reply to your first comment. I agree to the limited value of the enum case. You understanding of how the Assume: package sqlenum
type Enum[T comparable, K any] T
func (e Enum[T,K]) Value() (driver.Value, error) {
// Assume `enum.Lookup[T](k)` will check a registry to find an `EnumLookup` for type `T` with key `k`;
// and either panic or return an empty lookup when not found.
var k K
v, ok := enum.Lookup[T](k).Name(e)
if !ok {
return nil, fmt.Errorf("val")
}
}Then an type myKey struct{}
func init() {
// Assume MustRegister with register a loookup for `myKey{}` in the `enum` package or panic.
enum.MustRegister[MyType](myKey{}, enum.MustCreateLookup[MyType](map[MyType]string{
MyTypeValue1: "value1",
MyTypeValue2: "value2",
}))
}
type MyType = sqlenum.Enum[uint, myKey]
}This wouldn't become any better with Edit: for the case of enums, you could even use the (zero-value) of the Enum type itself as a key. package sqlenum
type Enum[T comparable] T
func (e Enum[T]) Value() (driver.Value, error) {
// Assume `enum.LookupFor[T]` will check a registry to find an `EnumLookup` for type `T` with the
// zero-value of `T` as a key.
v, ok := enum.LookupFor[Enum[T]]().Name(e)
if !ok {
return nil, fmt.Errorf("val")
}
}func init() {
// Assume MustRegister with register a loookup for `myKey{}` in the `enum` package or panic.
enum.MustRegisterLookup[MyType](
MyTypeValue1: "value1",
MyTypeValue2: "value2",
})
}
type MyType = sqlenum.Enum[uint]The code for the |
|
This is a complex set of ideas and everybody seems to have trouble understanding it. The need for this is unclear. It's not clear what important problem this solves that could be worth this additional complexity. For these reasons, and based on the above discussion, this is a likely decline. Leaving open for four weeks for final comments. |
|
Closing it sounds like the right choice. I will give some closing comments for my point of view. Thanks for good questions & comments. The initial use-case proposed in the issue (enum translation), turned out to be a bad one. At least for us, we where able to come up with a good alternative implementation that wouldn't benefit from any language changes. Discussion of enums on the issue comes to a similar conclusion; placing this information on the type is probably a symptom of the design. A place where I later thought the proposal to still have some merit to us, is for the encapsulated field use-case. The linked example shows an encapsulated regex field; we do have more encapsulated field cases where a I have no indication that encapsulated fields is a common pattern in Go. I think for this language feature to be considered, it would need multiple good use-cases with well-understood gains. |
|
No change in consensus. |
Go Programming Experience
Experienced
Other Languages Experience
C, Python
Related Idea
Has this idea, or one like it, been proposed before?
Yes
Typed enum support has been suggested in #19814. That proposal has a overlapping use-case, which is to allow conversion to/from enumerated values and strings.
This proposal is different in that it's not limited to enums; it's about type parameterization using values. This proposal could be used to define an
enumpackage (in or outside) the standard library to aid with enum conversion and validation, but it's not primarily a proposal for implementing enums.Const parameters has been suggested in #65555. That proposal aims at allowing to create types using constant parameters (as an alternative or addition to typed parameters).
This proposal is different, and perhaps arguably worse, in that it isn't limited to constant parameters. The proposal is also different in the sense that it's aiming for a clear separation between type and value parameters.
Does this affect error handling?
No
Is this about generics?
Yes, it's about allowing type initialization to refer to a value.
Proposal
Sometimes, it would be useful to initialize a type that is initialized not only using type parameters, but also value parameters. This allow generic code to be written for multiple use-cases, such as:
TableName() stringthat return a static result.sql.Scaneranddriver.Valuerimplementations refering to a type parameterized translation lookup. In the context of an API model, implementencoding.TextMarshallerandencoding.TextUnmarshallerinstead.Enum use-case
Goal: define an enum like value with minimal boiler plate (library code is allowed).
Current solution (no language change)
Library:
Application:
Suggested solution (language change)
Library:
Application:
Matrix use-case
An attempt of supporting what's described in #65555. There could be reasons in which this is harder than the first case. I am not sure it's a priority use-case.
Current solution (not type safe)
Solution with language change (type safe)
Alternatively, if it's easier for implementation reasons:
Language Spec Changes
To be decided; early draft
Type value parameter declarations
A type value parameter list declares the type value parameters of a generic type declaration; it can not be used by generic functions. The type value parameter list looks like an ordinary function parameter list except that the type parameter names must all be present and the list is enclosed in square brackets rather than parentheses. The declaration must come right after a type parameter list. If there are no type parameters, then an empty set of square brackets must precede the value parameter list.
Example (with a type parameter list prefix):
[T any][v T]
[][v string]
[][_ any]
Just as each ordinary function parameter has a parameter type, each type value parameter has a type.
Informal Change
Just like a function can have both input parameters and output parameters, a generic type can have both type and type value parameters. Both parameters are used to generate a concrete implementation. Type parameters are replaced by specific types, while type value parameters are replaces by a reference to a specific variable reference. That is, if you where to generate code for it, you would typically see that instances of the type parameters are replaced by the proper type reference, while references to values, are replaced by references to package scoped variables.
A simple (and somewhat silly) example:
Is this change backward compatible?
Yes
Orthogonality: How does this change interact or overlap with existing features?
It interacts with generic type declarations, by declaring a second parameter group.
Would this change make Go easier or harder to learn, and why?
I don't think it would make Go easier to learn. It's going to make it somewhat harder to learn generics in particular.
It might help deal with some pain-points if combined with the right standard libraries, such as easier enum handling.
Cost Description
Changes to Go ToolChain
Yes
Performance Costs
Unknown.
Prototype
I belie a ad-hock implementation could be written that depend on code generation.
Something similar to what was used for go generics, using
.go2or another file type format, could be used to scan the code and replace type value parameters with specific type implementations that replace the variable reference with a reference to a private package scoped variable; potentially one with a generated name.I.e.
would generatesomething like:
The proto-type could be limited not to allow usage of the new generics across package borders, and could run before compilation. It could also require explicit use of type aliases for declaring types.
A final implementations should not have such limitaitons.
The text was updated successfully, but these errors were encountered: