-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
database/gdb: enhance the underlying data transformation when querying data #3557
base: master
Are you sure you want to change the base?
Conversation
代码量有点多,需要花点时间review。 |
The amount of code is a bit large and it will take some time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
补充一下公共方法的注释
@@ -1621,6 +1622,13 @@ func Test_Types(t *testing.T) { | |||
Bit int8 | |||
TinyInt bool | |||
} | |||
now, err := db.Query(ctx, "select now();") | |||
t.Log("mysql now=", now, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
为啥要在单测里打日志,下面也一样
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
为啥要在单测里打日志,下面也一样
这个当然是有原因的,可以看issue #3558 关于gtime的bug,等下就把他删掉
} | ||
|
||
func Test_Core_Convert_Custom(t *testing.T) { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
函数开头结尾为啥要带空行,有哪个标准是这样做的吗
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
函数开头结尾为啥要带空行,有哪个标准是这样做的吗
可以有,也可以开启golangci-lint 检测拦截这种处理
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
可以有也行,保持一致就行
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
可以有也行,保持一致就行
原来是不留空行的,建议删除了
if i > 0 { | ||
if v.Kind() == reflect.Pointer { | ||
if v.IsNil() { | ||
v.Set(reflect.New(v.Type().Elem())) | ||
} | ||
v = v.Elem() | ||
} | ||
} | ||
v = v.Field(x) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
控制反转一下这边的代码,层级太深了
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
控制反转一下这边的代码,层级太深了
ok
if len(c.StructFieldIndex) == 1 { | ||
return structValue.Field(c.StructFieldIndex[0]) | ||
} | ||
v := structValue |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
为啥用变量再赋值一次
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
为啥用变量再赋值一次
这里当时偷懒了,因为这个方法的逻辑是从标准库cv过来的
database/gdb/gdb_model_select.go
Outdated
if ok { | ||
// It needs to be deleted, otherwise it will cause | ||
// conflicts as long as the struct name is the same within different functions during testing | ||
defer tablesMap.Delete(tableName) | ||
|
||
table = tableValue.(*Table) | ||
structPointerValue := reflect.ValueOf(pointer).Elem() | ||
|
||
var structValue = reflect.New(elemType) | ||
// UnmarshalValue | ||
fn, ok := structValue.Interface().(iUnmarshalValue) | ||
if ok { | ||
err = fn.UnmarshalValue(one) | ||
if err != nil { | ||
return err | ||
} | ||
structValue = structValue.Elem() | ||
} else { | ||
structValue = structValue.Elem() | ||
for _, field := range table.fields { | ||
fieldValue := field.GetReflectValue(structValue) | ||
value := one[field.ColumnFieldName] | ||
if value == nil { | ||
continue | ||
} | ||
fieldValue.Set(reflect.ValueOf(value.Val())) | ||
} | ||
} | ||
if elemIsPtr { | ||
structValue = structValue.Addr() | ||
} | ||
structPointerValue.Set(structValue) | ||
} else { | ||
if err = one.Struct(pointer); err != nil { | ||
return err | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if !ok {return}
....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@oldme-git 这里不能通过提前判断!ok 来返回,因为不管ok为true或者是false,都需要调用model.doWithScanStruct,来判断是否需要关联查询
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
在 !ok 中提前返回不可以吗
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
在 !ok 中提前返回不可以吗
不行,因为最后需要检查是否需要with关联查询,不过我已经把这个逻辑单独提取出来一个方法了,这样就可以提前返回了
添加注释ok |
Add comment ok |
@oldme-git 原来的类型转换依赖于gconv,gconv的转换过于复杂,从数据库的数据到结构体,不需要那么复杂的转换。此次pr的代码看起来可能比较多,其实就两部分,一部分是解析结构体字段,拿到字段的tag,索引,字段类型之类的,缓存起来,还有一部分就是具体的转换函数,这部分是最多的,但也是逻辑最简单的,就是挨个判断类型,然后匹配赋值即可 |
@oldme-git The original type conversion relies on gconv. The conversion of gconv is too complicated. From database data to structure, there is no need for such complicated conversion. The PR code this time may seem like a lot, but it actually consists of two parts. One part is to parse the structure fields, get the field tags, indexes, field types, etc., and cache them, and the other part is the specific conversion function. This It has the most parts, but it is also the simplest in logic. It is to judge the types one by one, and then match and assign values. |
改动确实有点多,需要多花些时间来 |
There are indeed a lot of changes, so it will take more time to |
- RegisterDatabaseConvertFunc 注册某个驱动(比如mysql)的数据库字段类型 - RegisterStructFieldConvertFunc 注册某个go语言结构体的字段 2. 一些细节优化
2.更新自定义转换的测试逻辑
本次改动只修改了Scan方法的底层的类型转换实现,会走新的逻辑,依旧是先把数据转换到map,然后在转到结构体上,
和之前的行为保持一致,不会导致不兼容。
从map到结构体的转换和之前不同的是,由于在底层转换时已经转成了具体类型,不需要调用gconv来做转换,可以直接使用循环赋值
====更新7.5
对于本次更新增加一个开关,打开时走新的逻辑,关闭时还是走旧逻辑
新增加自定义字段转换的特性,方便用户平稳过渡
1.RegisterDatabaseConvertFunc 为某个驱动(比如mysql)的某个类型(比如decimal)注册自定义转换函数
2.RegisterGoTypeConvertFunc为go语言的类型添加自定义转换函数,且优先级比1高
未修改之前的
修改后的
性能方面大幅提升,如果是直接一步赋值到结构体上,只需要600ms,但这样会导致有些api不兼容
一些建议
和orm的转换有关的接口,应当实现Scan和Value这两个接口来做,和标准库保持一致
建议废弃掉All,One之类的方法,一点都不好用,如果查询的是单个字段,或者是一个[]int,还需要再次转换才能使用,不如直接把Scan方法加强,支持 int , []int 这种类型的参数,一步赋值到目标字段,而不是走中间变量来回重复的转换
关于HookFuncSelect 的Next方法的返回值,不应该把原始的数据结构暴漏出去让用户操作,应当封装到结构体内部,通过方法去操作