Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

infoschema,planner: implement the CLUSTER_LOG memory table predicate push down #14018

Merged
merged 18 commits into from
Dec 16, 2019
Merged

infoschema,planner: implement the CLUSTER_LOG memory table predicate push down #14018

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
74bd94b
implement the CLUSTER_LOG memory table predicate push down
lonng Dec 10, 2019
c6322db
change multiple pattern protocol
lonng Dec 11, 2019
c602fda
Update planner/core/memtable_predicate_extractor.go
lonng Dec 11, 2019
d508a1b
update sysutil
lonng Dec 11, 2019
98be3ad
Merge remote-tracking branch 'origin/master' into cluster-log-push-down
lonng Dec 11, 2019
02a3d27
fix CI
lonng Dec 11, 2019
a49f2cb
Merge remote-tracking branch 'origin/master' into cluster-log-push-down
lonng Dec 11, 2019
ad8d1b1
address comment
lonng Dec 11, 2019
1395624
add some comments
lonng Dec 11, 2019
aae31ee
fix time fraction
lonng Dec 11, 2019
022211d
Merge remote-tracking branch 'origin/master' into cluster-log-push-down
lonng Dec 11, 2019
f7df50c
Merge branch 'master' into cluster-log-push-down
crazycs520 Dec 12, 2019
00fe80b
address comment
lonng Dec 14, 2019
6321029
Merge branch 'cluster-log-push-down' of github.com:lonng/tidb into cl…
lonng Dec 14, 2019
e993168
fix time range extraction
lonng Dec 14, 2019
f9d1d17
Merge branch 'master' into cluster-log-push-down
lonng Dec 14, 2019
861b205
Merge branch 'master' into cluster-log-push-down
crazycs520 Dec 16, 2019
0047748
Merge branch 'master' into cluster-log-push-down
crazycs520 Dec 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comment 
Signed-off-by: Lonng <heng@lonng.org>
  • Loading branch information
@lonng
lonng committed Dec 14, 2019
commit 00fe80b9b9dde043a51111fcdc7f2ba9bbff1d61
38 changes: 15 additions & 23 deletions planner/core/memtable_predicate_extractor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,13 +175,7 @@ func (helper extractHelper) extractCol(
) {
remained = make([]expression.Expression, 0, len(predicates))
result = set.NewStringSet()
extractCols := make(map[int64]*types.FieldName)
for i, name := range names {
if name.ColName.L == extractColName {
extractCols[schema.Columns[i].UniqueID] = name
}
}

extractCols := helper.findColumn(schema, names, extractColName)
if len(extractCols) == 0 {
return predicates, false, result
}
Expand Down Expand Up @@ -230,13 +224,7 @@ func (helper extractHelper) extractLikePatternCol(
patterns []string,
) {
remained = make([]expression.Expression, 0, len(predicates))
extractCols := make(map[int64]*types.FieldName)
for i, name := range names {
if name.ColName.L == extractColName {
extractCols[schema.Columns[i].UniqueID] = name
}
}

extractCols := helper.findColumn(schema, names, extractColName)
if len(extractCols) == 0 {
return predicates, nil
}
Expand All @@ -255,7 +243,7 @@ func (helper extractHelper) extractLikePatternCol(
var datums []types.Datum

switch fn.FuncName.L {
case ast.GT, ast.GE, ast.LT, ast.LE, ast.EQ, ast.Like, ast.Regexp:
case ast.EQ, ast.Like, ast.Regexp:
colName, datums = helper.extractColBinaryOpConsExpr(extractCols, fn)
}
if colName == extractColName {
Expand All @@ -276,6 +264,16 @@ func (helper extractHelper) extractLikePatternCol(
return
}

func (helper extractHelper) findColumn(schema *expression.Schema, names []*types.FieldName, colName string) map[int64]*types.FieldName {
extractCols := make(map[int64]*types.FieldName)
for i, name := range names {
if name.ColName.L == colName {
extractCols[schema.Columns[i].UniqueID] = name
}
}
return extractCols
}

// extracts the time range column, e.g:
// SELECT * FROM t WHERE time='2019-10-10 10:10:10'
// SELECT * FROM t WHERE time>'2019-10-10 10:10:10' AND time<'2019-10-11 10:10:10'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you don't deal with (time > ... and time < ...) or (time > ... and time < ...)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, to simplify the implementation of the diagnostics service search log interface, it can just accept only one range of time (pick a group of the log files in the time range). So the filtering will be executed in TiDB via Selection executor.

Copy link
Contributor Author

@lonng lonng Dec 14, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, we use the cluster level log search function to locate the problem usually only one range of time. So I think it's enough to cover this scenario.

Expand All @@ -292,13 +290,7 @@ func (helper extractHelper) extractTimeRange(
endTime int64,
) {
remained = make([]expression.Expression, 0, len(predicates))
extractCols := make(map[int64]*types.FieldName)
for i, name := range names {
if name.ColName.L == extractColName {
extractCols[schema.Columns[i].UniqueID] = name
}
}

extractCols := helper.findColumn(schema, names, extractColName)
if len(extractCols) == 0 {
return predicates, startTime, endTime
}
Expand All @@ -313,7 +305,7 @@ func (helper extractHelper) extractTimeRange(
var colName string
var datums []types.Datum
switch fn.FuncName.L {
case ast.GT, ast.GE, ast.LT, ast.LE, ast.EQ, ast.Like, ast.Regexp:
case ast.GT, ast.GE, ast.LT, ast.LE, ast.EQ:
colName, datums = helper.extractColBinaryOpConsExpr(extractCols, fn)
}

Expand Down