Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add week number to Elasticsearch output #3490

Merged
merged 1 commit into from
Nov 20, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion plugins/outputs/elasticsearch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ This plugin will format the events in the following way:
# %m - month (01..12)
# %d - day of month (e.g., 01)
# %H - hour (00..23)
# %V - week of the year (ISO week) (01..53)
index_name = "telegraf-%Y.%m.%d" # required.

## Optional SSL Config
Expand Down Expand Up @@ -220,6 +221,6 @@ Integer values collected that are bigger than 2^63 and smaller than 1e21 (or in

```{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_state_exception","reason":"No matching token for number_type [BIG_INTEGER]"}},"status":400}```

The correct field mapping will be created on the telegraf index as soon as a supported JSON value is received by Elasticsearch, and subsequent insertions will work because the field mapping will already exist.
The correct field mapping will be created on the telegraf index as soon as a supported JSON value is received by Elasticsearch, and subsequent insertions will work because the field mapping will already exist.

This issue is caused by the way Elasticsearch tries to detect integer fields, and by how golang encodes numbers in JSON. There is no clear workaround for this at the moment.
16 changes: 12 additions & 4 deletions plugins/outputs/elasticsearch/elasticsearch.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,16 @@ package elasticsearch
import (
"context"
"fmt"
"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/plugins/outputs"
"gopkg.in/olivere/elastic.v5"
"log"
"net/http"
"strconv"
"strings"
"time"

"github.com/influxdata/telegraf"
"github.com/influxdata/telegraf/internal"
"github.com/influxdata/telegraf/plugins/outputs"
"gopkg.in/olivere/elastic.v5"
)

type Elasticsearch struct {
Expand Down Expand Up @@ -58,6 +59,7 @@ var sampleConfig = `
# %m - month (01..12)
# %d - day of month (e.g., 01)
# %H - hour (00..23)
# %V - week of the year (ISO week) (01..53)
index_name = "telegraf-%Y.%m.%d" # required.

## Optional SSL Config
Expand Down Expand Up @@ -301,6 +303,7 @@ func (a *Elasticsearch) GetIndexName(indexName string, eventTime time.Time) stri
"%m", eventTime.UTC().Format("01"),
"%d", eventTime.UTC().Format("02"),
"%H", eventTime.UTC().Format("15"),
"%V", getISOWeek(eventTime.UTC()),
)

indexName = dateReplacer.Replace(indexName)
Expand All @@ -310,6 +313,11 @@ func (a *Elasticsearch) GetIndexName(indexName string, eventTime time.Time) stri

}

func getISOWeek(eventTime time.Time) string {
_, week := eventTime.ISOWeek()
return strconv.Itoa(week)
}

func (a *Elasticsearch) SampleConfig() string {
return sampleConfig
}
Expand Down
5 changes: 5 additions & 0 deletions plugins/outputs/elasticsearch/elasticsearch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,11 @@ func TestGetIndexName(t *testing.T) {
"indexname-%y-%m",
"indexname-14-12",
},
{
time.Date(2014, 12, 01, 23, 30, 00, 00, time.UTC),
"indexname-%Y-%V",
"indexname-2014-49",
},
}
for _, test := range tests {
indexName := e.GetIndexName(test.IndexName, test.EventTime)
Expand Down