Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

分库分表:分片结果不符预期修复 #174

Merged
merged 9 commits into from
Mar 26, 2023
Merged

Conversation

heroyf
Copy link
Contributor

@heroyf heroyf commented Mar 22, 2023

  • fix: hash和shadow_hash 分片在对非分片键查询时返回为空的问题修复

  • feat: NOT支持(暂不在在合并中支持)

Copy link
Collaborator

@Stone-afk Stone-afk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

老哥,"github.com/gotomicro/ekit/slice" 这个包的引用需要将 gotomicro 改成 ecodeclub

@heroyf
Copy link
Contributor Author

heroyf commented Mar 23, 2023

老哥,"github.com/gotomicro/ekit/slice" 这个包的引用需要将 gotomicro 改成 ecodeclub

v0.0.6 module 声明的还是gotomicro,我的理解是这里不做replace,等ekit下次版本迭代更名后,统一在go.mod中做replace
目前看还有很多其他地方import的是gotomicro

@flycash
Copy link
Contributor

flycash commented Mar 23, 2023

应该全部都改成了 ecodeclub,你试试更新代码?

@flycash
Copy link
Contributor

flycash commented Mar 23, 2023

哦哦,是 ekit 的啊。ekit 的暂时保留 gotomicro。要等我发布新的 ekit 版本之后再来切换。

sharding_select_test.go Show resolved Hide resolved
sharding_select_test.go Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Mar 23, 2023

Codecov Report

Merging #174 (2679e84) into dev (8996c2e) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              dev     #174      +/-   ##
==========================================
- Coverage   81.46%   81.31%   -0.15%     
==========================================
  Files          32       32              
  Lines        2600     2569      -31     
==========================================
- Hits         2118     2089      -29     
+ Misses        403      402       -1     
+ Partials       79       78       -1     
Impacted Files Coverage Δ
sharding_select.go 54.24% <100.00%> (-3.38%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

增加NOT-AND,NOT-OR,NOT-AND-OR单测用例
Copy link
Contributor

@flycash flycash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

暂停一下这个东西。我发现我的逻辑是错误的,对于 NOT 来说,NOT(C) 并不是广播减去命中的节点。
我举个例子,在 %2 分表的情况下,NOT(user_id = 123) 并不是说只查找 user_tab_0,而是 user_tab_1 都要查询,因为 user_id =125 符合条件,但是和 user_id = 123 在同一张表里面。

所以,某分库分表中间件对 NOT 的支持,实际上是广播=。=。

我们可以在这里讨论一下,如果 NOT 保持只广播,有没有办法优化?毕竟广播实在太差了

@flycash
Copy link
Contributor

flycash commented Mar 24, 2023

对于哈希来说,是广播;但是对于范围分库分表来说,则不是广播,这种算法可以通过对范围的上界和下界进行取反之后计算得到。

@flycash
Copy link
Contributor

flycash commented Mar 24, 2023

@heroyf 这个合并请求你暂时只修复一下那个 BUG,NOT 查询的话,可能要重新设计

@heroyf
Copy link
Contributor Author

heroyf commented Mar 24, 2023

对于哈希来说,是广播;但是对于范围分库分表来说,则不是广播,这种算法可以通过对范围的上界和下界进行取反之后计算得到。

想了一下这个,如果对于范围分库分表,是不是依旧还是广播,而不是简单对于上界下界取反?
例如按照UserId的范围进行区分(某一种分片算法实现>=500在table_2,<500在table_1),那么NOT(user_id = 510)也是需要查询两张表。所以对于NOT目前这两种情况都是广播,最终要解决的就是广播情况下的查询优化问题

这里可能还需要再深入考虑一下是否还有其他场景

@heroyf
Copy link
Contributor Author

heroyf commented Mar 24, 2023

@heroyf 这个合并请求你暂时只修复一下那个 BUG,NOT 查询的话,可能要重新设计

好的,这个我先修复对于非分片键等值查询结果为空的情况

@heroyf heroyf changed the title 分库分表:分片结果不符预期修复以及NOT特性支持 分库分表:分片结果不符预期修复 Mar 25, 2023
@flycash flycash linked an issue Mar 26, 2023 that may be closed by this pull request
@flycash flycash merged commit 508dbc1 into ecodeclub:dev Mar 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

分库分表hash、shadow_hash算法不符合预期
3 participants