Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

queue: able to set max disk used bytes for diskQueue #706

Merged
merged 1 commit into from
Aug 15, 2018
Merged

queue: able to set max disk used bytes for diskQueue #706

merged 1 commit into from
Aug 15, 2018

Conversation

unknwon
Copy link
Contributor

@unknwon unknwon commented Aug 13, 2018

JIRA: https://jira.qiniu.io/browse/PDR-7002

  1. 支持给 diskQueue 设置最大磁盘使用量,多余的数据直接丢弃
  2. queue 相关测试改为并行执行

queue/disk.go Outdated
opts.MemoryQueueSize = 100
}

if opts.EnableDiskUsedLimit && opts.MaxDiskUsedBytes <= 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

磁盘大小限制默认开启,默认的限制给到32GB

@redHJ
Copy link
Collaborator

redHJ commented Aug 14, 2018

lgtm

maxMemoryLength = 0
} else if enableMemory && maxMemoryLength <= 0 {
maxMemoryLength = 100
func NewDiskQueue(opts NewDiskQueueOptions) BackendQueue {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这样改这个包就不兼容了,到时候smartelf和logkit-pro升级要注意下

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本来就要多个参数编译就过不了的,一次性改成这种形式的以后更加方便添加更多选项

queue/disk.go Outdated
d.writeOne(msg)
err := d.writeOne(msg)
if err != nil {
log.Errorf("DISKQUEUE(%s): drop one msg from memory chan - %v", d.name, err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

理论上这个错误永远不会出现

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个可以再放进memory queue里面吗

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

目前唯一调用 saveToDisk 的地方是在 ioLoop 退出的时候:

logkit/queue/disk.go

Lines 806 to 808 in d044334

if d.enableMemoryQueue {
d.saveToDisk()
}

就目前的实现来讲,放回去也是没有意义的

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

那感觉这种情况应该强行放进去,否则丢数据了

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果 writeOne 一直失败就会无限循环退出不来

Copy link
Contributor Author

@unknwon unknwon Aug 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果 writeOne 一直失败就会无限循环退不出来

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不过之前就一直在丢数据... 这要怎么操作

logkit/queue/disk.go

Lines 793 to 800 in d044334

case <-d.exitChan:
if origin == FromMemory {
err = d.writeOne(dataRead)
if err != nil {
log.Errorf("DISKQUEUE(%s): drop one msg - %v", d.name, err)
}
}
break DONE

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你先加个FIXME吧,然后修一下CI

@wonderflow
Copy link
Contributor

LGTM

@wonderflow wonderflow merged commit 3406f3b into qiniu:master Aug 15, 2018
@unknwon unknwon deleted the PDR-7002-disk-queue-max-disk-usage branch August 15, 2018 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants