Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Telegraf can not handle big config files #745

Closed
titilambert opened this issue Feb 23, 2016 · 3 comments · Fixed by influxdata/toml#1
Closed

Telegraf can not handle big config files #745

titilambert opened this issue Feb 23, 2016 · 3 comments · Fixed by influxdata/toml#1
Labels
bug unexpected problem or unintended behavior

Comments

@titilambert
Copy link
Contributor

Hello !
As discuss with @sparrc, Telegraf ( using https://github.com/naoina/toml ) can not handle large file.
The limit seems to be : 32768 characters ... I made examples:

Next step : find where this is set (a int16 buffer ???)

@titilambert
Copy link
Contributor Author

I think I got it !

Apply this patch to https://github.com/naoina/toml

diff --git a/parse.peg.go b/parse.peg.go
index ce967e9..6c84510 100644
--- a/parse.peg.go
+++ b/parse.peg.go
@@ -238,7 +238,7 @@ type element struct {
 /* ${@} bit structure for abstract syntax tree */
 type token16 struct {
        pegRule
-       begin, end, next int16
+       begin, end, next int32
 }

 func (t *token16) isZero() bool {
@@ -299,7 +299,7 @@ func (t *tokens16) Order() [][]token16 {

        for i, token := range t.tree {
                depth := token.next
-               token.next = int16(i)
+               token.next = int32(i)
                ordered[depth][depths[depth]] = token
                depths[depth]++
        }
@@ -341,7 +341,7 @@ func (t *tokens16) PreOrder() (<-chan state16, [][]token16) {
                depths, state, depth := make([]int16, len(ordered)), 0, 1
                write := func(t token16, leaf bool) {
                        S := states[state]
-                       state, S.pegRule, S.begin, S.end, S.next, S.leaf = (state+1)%8, t.pegRule, t.begin, t.end, int16(depth), leaf
+                       state, S.pegRule, S.begin, S.end, S.next, S.leaf = (state+1)%8, t.pegRule, t.begin, t.end, int32(depth), leaf
                        copy(S.depths, depths)
                        s <- S
                }
@@ -456,7 +456,7 @@ func (t *tokens16) PrintSyntaxTree(buffer string) {
 }

 func (t *tokens16) Add(rule pegRule, begin, end, depth, index int) {
-       t.tree[index] = token16{pegRule: rule, begin: int16(begin), end: int16(end), next: int16(depth)}
+       t.tree[index] = token16{pegRule: rule, begin: int32(begin), end: int32(end), next: int32(depth)}
 }

 func (t *tokens16) Tokens() <-chan token32 {

@sparrc sparrc added the bug unexpected problem or unintended behavior label Feb 24, 2016
@titilambert
Copy link
Contributor Author

@sparrc Should we not change this line ?
https://github.com/influxdata/telegraf/blob/master/Godeps#L29
to use toml from influxdata ?

@sparrc
Copy link
Contributor

sparrc commented Feb 29, 2016

I didn't mean to close this, it's not fixed yet here

@sparrc sparrc reopened this Feb 29, 2016
@sparrc sparrc closed this as completed in 9af8d69 Mar 1, 2016
geodimm pushed a commit to miketonks/telegraf that referenced this issue Mar 10, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants