feature add sharding selector #145

Stone-afk · 2023-02-04T14:41:48Z

这是初步草稿思路：

Get那里确保只能生成一个 SQL，我理解是直接简单初爆的做判断？；
还有个人觉得表和库的sharding方法应该分开；
sharding 相关的功能是否应该独立成一个包？

codecov · 2023-02-04T14:43:18Z

Codecov Report

Merging #145 (d917273) into dev (08fc73f) will decrease coverage by 5.73%.
The diff coverage is 51.67%.

@@            Coverage Diff             @@
##              dev     #145      +/-   ##
==========================================
- Coverage   84.35%   78.63%   -5.73%     
==========================================
  Files          24       25       +1     
  Lines        1687     2045     +358     
==========================================
+ Hits         1423     1608     +185     
- Misses        226      377     +151     
- Partials       38       60      +22

Impacted Files	Coverage Δ
builder.go	`85.96% <ø> (ø)`
select.go	`90.17% <ø> (ø)`
sharding_select.go	`50.00% <50.00%> (ø)`
internal/model/model.go	`88.49% <100.00%> (+1.36%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

flycash · 2023-02-04T15:31:55Z

sharding 内部使用的放进去 internal 里面，如果是暴露给用户的接口，那么可以直接放在顶级目录下面，也可以在顶级目录下面创建一个 sharding 包；

分开其实也可以，只不过分开之后的设计可能会很像，因为理论上来说，任何可以被用来分库的算法，也可以用来分表。这个你随意，我不介意；

Get 那里你先直接粗暴判断，后面我们再来重构

flycash

思路是没什么问题的，你继续把 Build 写下去，然后补充单元测试。单元测试先试试 Build 方法就行。

sharding_select.go

…ding

Stone-afk · 2023-02-07T15:10:31Z

sharding selector 与 selector 存在大量的重复方法，这个是否需要共同的实现一个接口，以及添加一个基础的抽象如 BaseSelector, 不过泛型的话这里会很丑

flycash · 2023-02-12T08:50:44Z

sharding selector 与 selector 存在大量的重复方法，这个是否需要共同的实现一个接口，以及添加一个基础的抽象如 BaseSelector, 不过泛型的话这里会很丑

可以将一部分数据结构下沉到 internal 里面，而后 ShardingSelector 和普通的 Selector 共同依赖于这部分

flycash

全部注释什么的用中文。之前我用英文是因为那时候脑子进水了

sharding_db.go

sharding_select.go

sharding_select_test.go

codecov-commenter · 2023-02-25T06:25:30Z

Codecov Report

Merging #145 (6e8ed06) into dev (f57e3af) will decrease coverage by 5.36%.
The diff coverage is 59.66%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##              dev     #145      +/-   ##
==========================================
- Coverage   85.30%   79.95%   -5.36%     
==========================================
  Files          26       28       +2     
  Lines        1797     2265     +468     
==========================================
+ Hits         1533     1811     +278     
- Misses        219      376     +157     
- Partials       45       78      +33

Impacted Files	Coverage Δ
aggregate.go	`96.38% <ø> (ø)`
assignment.go	`66.66% <ø> (ø)`
builder.go	`85.96% <ø> (ø)`
column.go	`90.00% <ø> (ø)`
core.go	`95.16% <ø> (ø)`
db.go	`73.21% <ø> (ø)`
delete.go	`92.50% <ø> (ø)`
expression.go	`82.35% <ø> (ø)`
insert.go	`92.77% <ø> (ø)`
internal/dialect/dialect.go	`100.00% <ø> (ø)`
... and 18 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

flycash

我先合并了，它还很多 BUG，但是这算是我的锅，我应该拆分得更加细一点。不过说实在的，我也很难拆了=。=

flycash · 2023-02-25T12:58:06Z

sharding_db.go

+	return orm, nil
+}
+
+func ShardingDBOptionWithMetaRegistry(r model.MetaRegistry) ShardingDBOption {


不需要 Option 这个中缀，其它也是

flycash · 2023-02-25T12:59:28Z

sharding_db.go

+
+type ShardingDBOption func(db *ShardingDB)
+
+func CtxWithDBName(ctx context.Context, dbName string) context.Context {


这个暂时不要暴露出去。后面我们是要设计强制走某个库，某个表，到时候再说。其实我是很反对用户自己强制指定的，要么是他们代码有问题，要么是他们设计有问题

flycash · 2023-02-25T13:00:02Z

sharding_db.go

+	val := ctx.Value(dbNameKey{})
+	dbName, ok := val.(string)
+	if !ok {
+		return "", errs.ErrCtxGetDBName


ErrDBNotSpecified

flycash · 2023-02-25T13:02:23Z

sharding_select.go

+			return nil, err
+		}
+		res = append(res, query)
+		s.args = make([]any, 0, 8)


看起来有点奇怪，因为对应的 buffer 也是需要重置的

把它和 buffer 重置放一起，然后定义一个新的方法 reset()

flycash · 2023-02-25T13:05:03Z

sharding_select.go

+	if len(s.where) > 0 {
+		s.writeString(" WHERE ")
+		p := s.where[0]
+		for i := 1; i < len(s.where); i++ {
+			p = p.And(s.where[i])
+		}
+		if err = s.buildExpr(p); err != nil {
+			return nil, err
+		}
+	}
+
+	// group by
+	if len(s.groupBy) > 0 {
+		err = s.buildGroupBy()
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	// order by
+	if len(s.orderBy) > 0 {
+		err = s.buildOrderBy()
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	// having
+	if len(s.having) > 0 {
+		s.writeString(" HAVING ")
+		p := s.having[0]
+		for i := 1; i < len(s.having); i++ {
+			p = p.And(s.having[i])
+		}
+		if err = s.buildExpr(p); err != nil {
+			return nil, err
+		}
+	}
+
+	if s.offset > 0 {
+		s.writeString(" OFFSET ")
+		s.parameter(s.offset)
+	}
+
+	if s.limit > 0 {
+		s.writeString(" LIMIT ")
+		s.parameter(s.limit)
+	}
+	s.end()


之前我说过引入一个 selectorBuilder。然后把 Selector 和 ShardingSelector 共用的东西丢进去

flycash · 2023-02-25T13:23:07Z

sharding_db.go

+	return sess, nil
+}
+
+func (s *ShardingDB) broadcast() []Dst {


所以在广播的时候你会出现问题，就是比如说我这个数据库虽然分库分表了，但是我其实 Order 表和 OrderDetail 表都在这个库上，那么你这里的广播就会导致，原本我只查询 Order 表的，结果你会广播过去 OrderDetail。这也就是我说，应该定义在分库分表规则上的

flycash · 2023-02-25T13:26:30Z

sharding_select_test.go

+		},
+		{
+			name: "miss sharding key err",
+			builder: func() ShardingQueryBuilder {


你这个不是已经传入了 UserId 吗？

flycash · 2023-02-25T13:27:47Z

sharding_select_test.go

+						"order_tab_3": true,
+					}))
+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).Where(C("UserId").EQ(int64(123)))


这个 Selector 和前面一个测试用例不是一样的吗？

flycash · 2023-02-25T13:29:33Z

sharding_select_test.go

+				}, ShardingDBOptionWithMetaRegistry(r),
+					ShardingDBOptionWithTables(map[string]bool{
+						"order_tab_3": true,
+					}))
+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).
+					Select(Columns("Content", "OrderId")).Where(C("UserId").EQ(int64(123)))
+				return s
+			}(),
+			qs: []*ShardingQuery{
+				{
+					SQL:  "SELECT `content`,`order_id` FROM `order_db_1`.`order_tab_3` WHERE `user_id`=?;",
+					Args: []any{int64(123)},
+					DB:   "order_db_1",
+				},
+			},
+		},
+		{
+			name: "invalid columns",
+			builder: func() ShardingQueryBuilder {
+				shardingDB, err := OpenShardingDB("sqlite3", map[string]*MasterSlavesDB{
+					"order_db_1": masterSlaveDB,
+				}, ShardingDBOptionWithMetaRegistry(r),
+					ShardingDBOptionWithTables(map[string]bool{
+						"order_tab_3": true,
+					}))
+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).
+					Select(C("Invalid")).Where(C("UserId").EQ(int64(123)))
+				return s
+			}(),
+			wantErr: errs.NewInvalidFieldError("Invalid"),
+		},
+		{
+			name: "order by",
+			builder: func() ShardingQueryBuilder {
+				shardingDB, err := OpenShardingDB("sqlite3", map[string]*MasterSlavesDB{
+					"order_db_1": masterSlaveDB,
+				}, ShardingDBOptionWithMetaRegistry(r),
+					ShardingDBOptionWithTables(map[string]bool{
+						"order_tab_3": true,
+					}))
+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).Select(C("OrderId"), C("Content")).
+					Where(C("UserId").EQ(int64(123))).OrderBy(ASC("UserId"), DESC("OrderId"))
+				return s
+			}(),
+			qs: []*ShardingQuery{
+				{
+					SQL:  "SELECT `order_id`,`content` FROM `order_db_1`.`order_tab_3` WHERE `user_id`=? ORDER BY `user_id` ASC,`order_id` DESC;",
+					Args: []any{int64(123)},
+					DB:   "order_db_1",
+				},
+			},
+		},
+		{
+			name: "group by",
+			builder: func() ShardingQueryBuilder {
+				shardingDB, err := OpenShardingDB("sqlite3", map[string]*MasterSlavesDB{
+					"order_db_1": masterSlaveDB,
+				}, ShardingDBOptionWithMetaRegistry(r),
+					ShardingDBOptionWithTables(map[string]bool{
+						"order_tab_3": true,
+					}))
+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).Select(C("OrderId"), C("Content")).
+					Where(C("UserId").EQ(int64(123))).GroupBy("UserId", "OrderId")
+				return s
+			}(),
+			qs: []*ShardingQuery{
+				{
+					SQL:  "SELECT `order_id`,`content` FROM `order_db_1`.`order_tab_3` WHERE `user_id`=? GROUP BY `user_id`,`order_id`;",
+					Args: []any{int64(123)},
+					DB:   "order_db_1",
+				},
+			},
+		},
+		{
+			name: "having",
+			builder: func() ShardingQueryBuilder {
+				shardingDB, err := OpenShardingDB("sqlite3", map[string]*MasterSlavesDB{
+					"order_db_1": masterSlaveDB,
+				}, ShardingDBOptionWithMetaRegistry(r),
+					ShardingDBOptionWithTables(map[string]bool{
+						"order_tab_3": true,
+					}))
+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).
+					Select(C("OrderId"), C("Content")).
+					Where(C("UserId").EQ(int64(123))).GroupBy("OrderId").Having(C("OrderId").EQ(int64(18)))
+				return s
+			}(),
+			qs: []*ShardingQuery{
+				{
+					SQL:  "SELECT `order_id`,`content` FROM `order_db_1`.`order_tab_3` WHERE `user_id`=? GROUP BY `order_id` HAVING `order_id`=?;",
+					Args: []any{int64(123), int64(18)},
+					DB:   "order_db_1",
+				},
+			},
+		},


这几个不是测试 WHERE 部分写法的，挪出去单独作为一个测试，因为它们不太重要=。=后面等我支持聚合函数的时候再考虑弄回来

flycash · 2023-02-25T13:30:36Z

sharding_select_test.go

+				require.NoError(t, err)
+				s := NewShardingSelector[Order](shardingDB).
+					Select(C("OrderId"), C("Content")).
+					Where(C("OrderId").EQ(int64(12)).And(C("UserId").EQ(int64(123))))


我在 opEQ 那里评价太苛刻就是类似这种用法，例如 C("OrderId").Eq(Raw("now()"))

feature add sharding selector

9a511b1

feature add fidDst

babaf7a

flycash requested changes Feb 7, 2023

View reviewed changes

sharding_select.go Outdated Show resolved Hide resolved

sharding_select.go Outdated Show resolved Hide resolved

sharding_select.go Outdated Show resolved Hide resolved

sharding_select.go Outdated Show resolved Hide resolved

Merge branch 'dev' of https://github.com/gotomicro/eorm into dev_shar…

3454d1b

…ding

feature add CHANGELOG

8954e9f

feature add integration test

0b0cfe8

Stone-afk changed the title ~~WIP: feature add sharding selector~~ feature add sharding selector Feb 15, 2023

Stone-afk added 5 commits February 16, 2023 09:05

feature merge dev

afc7c10

feature add test

4da7a72

feature add test

17ffbdd

feature add test

b19b3c8

feature add test

d917273

flycash requested changes Feb 17, 2023

View reviewed changes

Stone-afk added 6 commits February 24, 2023 22:28

feature add or

c40b72a

feature update repo

243d326

feature update repo

eb7f0cf

feature add coment

0b7e882

feature modify conflict

4e81982

feature modify conflict

6e8ed06

flycash approved these changes Feb 25, 2023

View reviewed changes

flycash merged commit 101df98 into ecodeclub:dev Feb 25, 2023

Stone-afk deleted the dev_sharding branch February 26, 2023 02:21

flycash linked an issue Feb 27, 2023 that may be closed by this pull request

分库分表：ShardingSelector 实现 #143

Closed

flycash mentioned this pull request Feb 27, 2023

ShardingSelector: 增强的 ShardingAlgorithm 设计与实现 #157

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature add sharding selector #145

feature add sharding selector #145

Stone-afk commented Feb 4, 2023 •

edited

Loading

codecov bot commented Feb 4, 2023 •

edited

Loading

flycash commented Feb 4, 2023

flycash left a comment

Stone-afk commented Feb 7, 2023

flycash commented Feb 12, 2023

flycash left a comment

codecov-commenter commented Feb 25, 2023

flycash left a comment

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023

flycash Feb 25, 2023


		type ShardingDBOption func(db *ShardingDB)

		func CtxWithDBName(ctx context.Context, dbName string) context.Context {

feature add sharding selector #145

feature add sharding selector #145

Conversation

Stone-afk commented Feb 4, 2023 • edited Loading

codecov bot commented Feb 4, 2023 • edited Loading

Codecov Report

flycash commented Feb 4, 2023

flycash left a comment

Choose a reason for hiding this comment

Stone-afk commented Feb 7, 2023

flycash commented Feb 12, 2023

flycash left a comment

Choose a reason for hiding this comment

codecov-commenter commented Feb 25, 2023

Codecov Report

flycash left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stone-afk commented Feb 4, 2023 •

edited

Loading

codecov bot commented Feb 4, 2023 •

edited

Loading