feature: Datasource 抽象 #170

Stone-afk · 2023-03-14T13:38:43Z

No description provided.

codecov · 2023-03-14T13:41:06Z

Codecov Report

Merging #170 (49f77a3) into dev (a97835a) will increase coverage by 0.69%.
The diff coverage is 75.47%.

@@            Coverage Diff             @@
##              dev     #170      +/-   ##
==========================================
+ Coverage   81.82%   82.51%   +0.69%     
==========================================
  Files          33       36       +3     
  Lines        2657     2699      +42     
==========================================
+ Hits         2174     2227      +53     
+ Misses        404      388      -16     
- Partials       79       84       +5

Impacted Files	Coverage Δ
...nal/datasource/masterslave/slaves/dns/mysql/dsn.go	`66.66% <ø> (ø)`
...source/masterslave/slaves/roundrobin/roundrobin.go	`86.20% <50.00%> (ø)`
...l/datasource/shardingsource/sharding_datasource.go	`76.92% <53.84%> (ø)`
sharding_select.go	`54.24% <58.82%> (ø)`
update.go	`82.70% <60.00%> (ø)`
select.go	`90.24% <63.63%> (ø)`
insert.go	`92.77% <66.66%> (ø)`
internal/datasource/masterslave/slaves/dns/dns.go	`76.78% <66.66%> (ø)`
internal/datasource/single/db.go	`71.42% <71.42%> (ø)`
delete.go	`92.50% <75.00%> (ø)`
... and 8 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

flycash

在 internal 下面直接创建一个 datasource。datasource 用于 sharding，但是并不归属于 sharding;
在 internal/datasource 里面定义接口
在 internal/datasource 里面提供三个实现：单一 DB，主从，Cluster DB
将已有的 slaves 的包挪过去对应的主从实现的子包里面

以下是要尝试一下，我预期中的：

让 eorm.DB 里面持有一个 datasource，而后在 eorm 层面上我们只有 DB 这个概念。也就是说，通过 eorm.DB 里面维持不同的 datasource 实现，那么它可以统一表达任意类型的 DB
在上一条成立的前提下，引入一个 OpenDS(ds datasource.Datasource) 的方法
在上一条成立的前提下，重构已有的代码，将 eorm 层面上对 datasource实现依赖的，都修改为对 DB 的依赖

我认为，在 ORM 层面上，应该不关心底层的 datasource 究竟是单一数据库，还是主从，还是分库分表的。虽然看起来难以彻底做到，但是至少朝着这里面去前进。

flycash · 2023-03-16T12:28:24Z

eorm.DB 内部维持一个 datasource 字段，然后让 DB 实现 session。 Session 接口组合你的 Datasource 接口

Stone-afk · 2023-03-16T12:42:04Z

这么搞普通的查询用不了主从数据库了，因为都必须实现
‘’‘
type DataSource interface {
Query(ctx context.Context, query *sharding.Query) (*sql.Rows, error)
Exec(ctx context.Context, query *sharding.Query) (sql.Result, error)
}
’‘’
必须要sharding才能使用

flycash · 2023-03-16T12:43:12Z

所以我说有一个普通的 DB 对应的 Datasource 实现。总共三个

flycash · 2023-03-16T12:43:36Z

datasource.DB 和 eorm.DB，你不要搞混了，这个是两个。后者是前者的装饰器

flycash · 2023-03-16T12:44:04Z

应该说，eorm.DB 是 datasource.Datasource 的装饰器，提供了 core 的那个功能

Stone-afk · 2023-03-16T13:07:44Z

对于事物的实现，我理解应该说在 erom.DB 里维护？但集群怎么保证事物呢

Stone-afk · 2023-03-16T13:10:22Z

还是说只要保证 sharding 后的目标主库写入数据就行，还有这么一来是否非分库分表就不能使用集群cluster

flycash · 2023-03-16T13:38:15Z

主从和单一的，你直接搞，和正常的事务没什么区别。分库分表的，这个要另外开 issue。分库分表的事务是一个跟棘手的事情，比这个 Datasource 棘手

Stone-afk · 2023-03-18T07:27:09Z

'''
type Query interface {
Sql() string
Args() []any
}
'''
通过统一制定 Query 然后在 eorm 与 sharding 里提供具体的实现，然后在cluster 里通过类型断言区分 eorm 与 sharding 的具体实现，获得额外的参数 DB 与 Datasource

flycash · 2023-03-23T13:11:18Z

我草！46 个文件，要命了！

flycash

我明天试试能不能删掉 cluster_db，看上去你这里没有用到。

internal/datasource/single/db.go

internal/datasource/cluster/cluster_db.go

internal/datasource/types.go

internal/datasource/single/db.go

internal/datasource/slaves/master_slave_db.go

internal/datasource/transaction/transaction.go

middleware/querylog/querylog_test.go

Stone-afk · 2023-03-25T16:58:24Z

@flycash

internal/datasource/cluster/cluster_db.go

internal/datasource/transaction/transaction.go

internal/datasource/types.go

internal/datasource/transaction/transaction.go

internal/datasource/single/db.go

internal/datasource/shardingsource/sharding_datasource.go

internal/datasource/transaction/transaction_test.go

internal/datasource/masterslave/slaves/roundrobin.go

Stone-afk · 2023-03-27T11:17:21Z

@flycash

builder.go

db.go

internal/sharding/types.go

middleware.go

middleware_test.go

transaction.go

types.go

Stone-afk

mark

flycash · 2023-03-29T14:55:36Z

你的 Golang-CI 崩了呀！

Stone-afk · 2023-03-30T14:31:12Z

ci lint 和排序的那个pr的合入有很大关系，要问 zwl 同学了

longyue0521 · 2023-03-30T14:42:26Z

ci lint 和排序的那个pr的合入有很大关系，要问 zwl 同学了

你更新一下你的pre-push/pre-commit及setup.sh, golangci-lint版本之前有问题所以干脆和github action一样用@latest版本,重新安装一下golangci-lint

Stone-afk · 2023-03-30T15:33:14Z

golangci-lint已经是最新版本了，并重新安装了

Stone-afk · 2023-03-30T15:34:04Z

juniaoshaonian · 2023-03-30T15:41:51Z

golangci-lint --version 看看版本呢，我之前換了最新的版本就行了，我的go版本是1.19的顺便这个也改一下

juniaoshaonian · 2023-03-30T15:49:15Z

你换 Go 1.19 了吗

是的

db_test.go

internal/datasource/cluster/cluster_db.go

internal/datasource/masterslave/slaves/dns/dns.go

internal/datasource/masterslave/slaves/roundrobin/roundrobin.go

internal/datasource/shardingsource/sharding_datasource_test.go

flycash · 2023-04-01T13:10:16Z

我一直觉得这个单元测试覆盖率简直有毒，不知道怎么算的

Stone-afk · 2023-04-02T09:54:36Z

@flycash

flycash

一个个你确定解决了的 conversation 你就点一下 resolve。不然你可能漏了...

db_test.go

flycash · 2023-04-03T06:56:27Z

internal/datasource/cluster/cluster_db.go

+	return ms.Exec(ctx, query)
+}
+
+//func (*clusterDB) BeginTx(_ context.Context, _ *sql.TxOptions) (datasource.Tx, error) {


不用的代码就删掉

internal/datasource/cluster/cluster_db.go

Stone-afk

mark

flycash · 2023-04-05T14:07:00Z

后续我再来微调

feature: Datasource 抽象

6af0719

flycash requested changes Mar 16, 2023

View reviewed changes

Stone-afk added 6 commits March 20, 2023 21:51

feature: datasource 实现

d01d71b

feature: datasource 实现

f06982e

feature: add test

de4cc35

feature: add test

b1d9941

feature: add test

6f8be6e

Merge branch 'dev' of https://github.com/ecodeclub/eorm into dev_164

f25c313

Stone-afk changed the title ~~WIP: Datasource 抽象~~ feature: Datasource 抽象 Mar 23, 2023

Stone-afk added 2 commits March 23, 2023 19:26

feature: merge dev

6f9b646

feature: merge dev

3f1337d

flycash reviewed Mar 23, 2023

View reviewed changes

internal/datasource/single/db.go Show resolved Hide resolved

flycash requested changes Mar 24, 2023

View reviewed changes

feature: add tx interface

1a7aabe

flycash requested changes Mar 26, 2023

View reviewed changes

Stone-afk added 3 commits March 27, 2023 18:57

feature: add pkg

6e80f1b

feature: add pkg

8dc1d6d

feature: motify test

33b0e44

flycash requested changes Mar 28, 2023

View reviewed changes

feature: datasource add close

cf6ca70

Stone-afk commented Mar 29, 2023

View reviewed changes

longyue0521 mentioned this pull request Mar 30, 2023

分库分表：Datasource相关总issue #186

Open

2 tasks

feature: modify DB

23f8569

feature: modify DB

30f22b3

feature: run lint

65bbb13

flycash requested changes Apr 1, 2023

View reviewed changes

feature: add test

0820420

ecodeclub deleted a comment from flycash Apr 3, 2023

flycash requested changes Apr 3, 2023

View reviewed changes

Stone-afk added 2 commits April 3, 2023 20:10

feature: add test

fe22dba

feature: add test

49f77a3

Stone-afk commented Apr 3, 2023

View reviewed changes

flycash approved these changes Apr 5, 2023

View reviewed changes

flycash merged commit 19f56ff into ecodeclub:dev Apr 5, 2023

Stone-afk deleted the dev_164 branch April 5, 2023 14:32

flycash mentioned this pull request Apr 8, 2023

分库分表：Datasource 抽象 #164

Closed

flycash linked an issue Apr 8, 2023 that may be closed by this pull request

分库分表：Datasource 抽象 #164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Datasource 抽象 #170

feature: Datasource 抽象 #170

Stone-afk commented Mar 14, 2023

codecov bot commented Mar 14, 2023 •

edited

Loading

flycash left a comment

flycash commented Mar 16, 2023

Stone-afk commented Mar 16, 2023

flycash commented Mar 16, 2023

flycash commented Mar 16, 2023

flycash commented Mar 16, 2023

Stone-afk commented Mar 16, 2023

Stone-afk commented Mar 16, 2023

flycash commented Mar 16, 2023

Stone-afk commented Mar 18, 2023

flycash commented Mar 23, 2023

flycash left a comment

Stone-afk commented Mar 25, 2023

Stone-afk commented Mar 27, 2023

Stone-afk left a comment

flycash commented Mar 29, 2023

Stone-afk commented Mar 30, 2023

longyue0521 commented Mar 30, 2023 •

edited

Loading

Stone-afk commented Mar 30, 2023

Stone-afk commented Mar 30, 2023

juniaoshaonian commented Mar 30, 2023 •

edited

Loading

juniaoshaonian commented Mar 30, 2023

flycash commented Apr 1, 2023

Stone-afk commented Apr 2, 2023

flycash left a comment

flycash Apr 3, 2023

Stone-afk left a comment

flycash commented Apr 5, 2023

feature: Datasource 抽象 #170

feature: Datasource 抽象 #170

Conversation

Stone-afk commented Mar 14, 2023

codecov bot commented Mar 14, 2023 • edited Loading

Codecov Report

flycash left a comment

Choose a reason for hiding this comment

flycash commented Mar 16, 2023

Stone-afk commented Mar 16, 2023

flycash commented Mar 16, 2023

flycash commented Mar 16, 2023

flycash commented Mar 16, 2023

Stone-afk commented Mar 16, 2023

Stone-afk commented Mar 16, 2023

flycash commented Mar 16, 2023

Stone-afk commented Mar 18, 2023

flycash commented Mar 23, 2023

flycash left a comment

Choose a reason for hiding this comment

Stone-afk commented Mar 25, 2023

Stone-afk commented Mar 27, 2023

Stone-afk left a comment

Choose a reason for hiding this comment

flycash commented Mar 29, 2023

Stone-afk commented Mar 30, 2023

longyue0521 commented Mar 30, 2023 • edited Loading

Stone-afk commented Mar 30, 2023

Stone-afk commented Mar 30, 2023

juniaoshaonian commented Mar 30, 2023 • edited Loading

juniaoshaonian commented Mar 30, 2023

flycash commented Apr 1, 2023

Stone-afk commented Apr 2, 2023

flycash left a comment

Choose a reason for hiding this comment

flycash Apr 3, 2023

Choose a reason for hiding this comment

Stone-afk left a comment

Choose a reason for hiding this comment

flycash commented Apr 5, 2023

codecov bot commented Mar 14, 2023 •

edited

Loading

longyue0521 commented Mar 30, 2023 •

edited

Loading

juniaoshaonian commented Mar 30, 2023 •

edited

Loading