latest master #1

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Enable tracing loki in helm chart Tracing support from #328, need to support config in helm chart Signed-off-by: Xiang Dai <764524258@qq.com> * Use camel case for variables Signed-off-by: Xiang Dai <764524258@qq.com> * update condition Signed-off-by: Xiang Dai <764524258@qq.com>

* Update troubleshooting - remove "no label" title - add tracing part Signed-off-by: Xiang Dai <764524258@qq.com> * Add helm help Signed-off-by: Xiang Dai <764524258@qq.com>

* Revert "Make sure the default for EnforceMetricName is ✅ (#518)" This reverts commit 199746a. * Fix overrides unmarshalling properly Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Add target config Support costomize target config. Signed-off-by: Xiang Dai <764524258@qq.com> * Update default value Signed-off-by: Xiang Dai <764524258@qq.com> * bump helm version Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

* support config chunk size Signed-off-by: Xiang Dai <764524258@qq.com> * fix lint Signed-off-by: Xiang Dai <764524258@qq.com> * review feedback

* Add chunk_block_size option Introduce by #464. Signed-off-by: Xiang Dai <764524258@qq.com> * fix white noise Signed-off-by: Xiang Dai <764524258@qq.com> * Bump up helm version Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

improve locking of positions

…ed up

…ers which wrap the binary with delve and allow for remote debugging

…r as it could be way in the past or future depending on the clock for the source system generating the logs. Given that it's compared to the servers time.Now() when looking for idle chunks it's probably best to use the server time when setting the value. (#546)

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* switch to golangci-lint linter * fixes for review comments

* add a custom server for loki with assets embedded * adds a targets page for promtail to see discovered files and labels * lint fixes * fix css table size * add service discovery logic * add service discovery template * add promtail pages documentation * ignored gen file in linter

* Adds the ability to provide multiple Loki URL For backward compatibility `client:` still works with flag. * add some tests for multi client * update ksonnet module to support multiple client * fix comment * fix lint issues

* fix backward compatibility with client config * fix comment

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

…eant the `pkg/promtail/server/ui/assets_vfsdata.go: assets` target is never activated causing a build error, updating to make the target hit on `server.go` which has a dependency on the deleted `assets_vfsdata.go` triggering a run of `make assets` before go builds that file (#555)

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Moved the check for already closed to make sure we honor this check when `immediate` flushing is requested

…k in a different branch

Co-authored-by: Edward Welch <edward.welch@grafana.com>

adding CRI and Docker format pipeline stage extensions adding backwards compatibility with current config

Updating docs Rebase fixes

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* add optional PodDisruptionBudget to helm chart * address comments * add back new line - no change to helpers template * fix silly typo in loki-stack version

…reate. Changed the timing of the sync function in the unit test to fix flakiness in the test.

…ine file, also changing unit to seconds and namespace to logentry

mostly working, need to extract out the config needs a proper ksonnet or helm deployment or both needs a makefile

improving tests config via flags added ksonnet config

* vendor: update cortex Don't be blocked on cortexproject/cortex#1345 to be merged and still maintain the fork Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * add memcached ksonnet Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * commit generated proto Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Add custom help info

Add doc about how to contribute helm

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

…he pod name from a file and the downardApi added a makefile

…onfigurable

* Limiting query start time with config * Fixed maxLookBackPeriod check, added it to existing configs, some other fixes * gofmted file * Changed maxLookBackPeriod to 4 weeks in helm chart * Setting maxLookBackPeriod in helm chart to 0 and bumping up the helm chart version * changed maxLookBackPeriod in yaml in cmd to 0 and some nit

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

* impr/logcli: Added label filters + tests * Address review * udpate readme

* impr/clients: Handle TLS config and MTLS for logcli and promtail * fix/tls: Please gofmt... * impr/clients: use prometheus HTTPClientConfig for logcli and promtail * fix/promtail: Set proper Client config name * impr/promtail: Use prometheus HTTPClientConfig configuration * adapt with master * address review * fix conflicts * address requested changes * remove file Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

) Changed it to dict with single key called "streams" and value set to list of logproto.Stream

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Fix local config: Use DayTime Psst, the date is the first commit date for loki ;) Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Fix the config in ksonnet and helm too Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Fix typo Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Small update to fix the link to the Cortex project.

…promtail chart versions are changed (#612)

* add a dev target to deploy the current image in k8s * impr/clients: Handle TLS config and MTLS for logcli and promtail (#540) * impr/clients: Handle TLS config and MTLS for logcli and promtail * fix/tls: Please gofmt... * impr/clients: use prometheus HTTPClientConfig for logcli and promtail * fix/promtail: Set proper Client config name * impr/promtail: Use prometheus HTTPClientConfig configuration * adapt with master * address review * fix conflicts * address requested changes * remove file * add helm dev targets * adding back assets * fix review comments * Review feedback

* Update cortex vendor Use query max look back from cortex Update config changes from cortex Fixed breaking code due to updating cortex vendor * fixed linter error

* ksonnet changes for running loki in single bianry Added retention config with default values * Fixed indentation

Mentioned in title. To save new beginner's time, provide one config example for use Cassandra as Index Storage.

* loki chart deployment.yaml: Only set JAEGER_AGENT_HOST if set. * docs/troubleshooting.md: Fixed variable name for chart option. * Bumped loki chart version for variable fix.

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

* Reject entries based on age set in limits

Still needs: Matcher stage work, pipeline test work

…plement the Match stage (and it cleans up the Docker and CRI extensions some too) Still needs to fix the metrics test

metric test is still failing because of missing metrics when the counter is 0 added a lot of TODO's I need to go back and cleanup

refactored and improved some of the tests, each stage type should now have a test using actual YAML A few TODO's remaining and some comment cleanup

cleaning up GoDoc pipeline_name is now optional for matcher stage

…similar to labels stage which is also plural. Adding a couple unit tests to regex and json stage to act as examples

removed helm entry in scrape_config.sh because there doesn't seem to be anyway to make it work with new pipeline config

…actually necessary)

Updates cortex vendor to get the changes from cortexproject/cortex#1431. Required for making promtail build on Windows Signed-off-by: sh0rez <me@shorez.de>

Build promtail on linux and windows, discard artifacts

Signed-off-by: Xiang Dai <764524258@qq.com>

* fix(promtail/targets): remove dependency on prometheus/relabel `filetargetmanager.go` had a dependency on prometheus/relabel, a package that has been removed in favor of prometheus/pkg/relabel. This converts the code to the new package, to allow the prometheus vendor be updated to master * chore(vendor): update prometheus vendor Updates prometheus vendor to current master, to add support for InitContainers in kubernetes service discovery * chore(vendor): do dep's homework Gives dep hints on how to resolve the vendor so that it work * fix(promtail/targets): non-nil check Accidentally checked against the wrong labelSet, this one can never get nil

Because of the logger being initialized before the configuration file is parsed, the log_level from the config file is ignored. To solve this, the logger is reinitialized after the file is parsed, as it is already being done in promtail.

…a direct query to loki to look for logs not received over the websocket before reporting them as missing

* Helm chart integration testing Signed-off-by: Steven Sheehy <ssheehy@firescope.com> * Don't check helm version on every push Signed-off-by: Steven Sheehy <ssheehy@firescope.com> * Remove chart upgrade testing Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

#626)

…n against entries which were not expected so that we can report them as duplicates.

* adds first version of docker driver * without logrus and fixes some linter issue * fix the driver and start a build system * adds swarm label discovery * Add documentation and more targets * make the linter happy ❤️ * indent config.json * with circleci steps for master and branch * Review Feebacks

* fix docker plugin ci * use binary cache

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

component/integrations: rename fluent plugin

…ost of my empirical testing we were using 9-10k out of the 32k allocated leaving about 2/3 of the buffer allocated and unused times the number of blocks per chunk, times the number of chunks kept in memory. This was adding up to a fair amount of allocated but unused space.

updating the loki dashboards to use the metrics from the new gateway

Signed-off-by: Xiang Dai <764524258@qq.com>

* Improvements in live tailing of logs Instead of polling for new logs, a grpc stream is opened between ingester and querier to get live logs Querier reconnects to disconnected or newly added ingesters * Added more comments to code for live log tailing * Some code refactoring in live log tailing * handling delayfor in logcli, max delayfor to be 5 seconds * some changes in tail response in live tailing * Fixed issue with stopping ingesters gracefully when live tailing is being used * Added tests for tailer in querier * Live tailing made logql compatible, some code refactoring suggested in PR

Signed-off-by: Xiang Dai <764524258@qq.com>

Remove useless config option

…second it would cause some way too aggressive querying when scaled out over all our clusters and loki had a hiccup

fix logcli code src path

…a `path` label, also removing the label values when no longer tailing the file. Following the same pattern we used for readBytes and totalBytes.

…r the log_entries_bytes histogram

Fixes #715

* fluent-plugin: Mark as multi-workers ready * fluent-plugin: Add info on multi-worker usage to README

When a more or less invalid config (e.g. `null`) is supplied, the log level prop of the config receives an invalid value which causes the logger to panic on re-init. Prevents this by checking for a nil log-level and notifies the user in case

Renames the `prometheus_rules` key to `prometheusRules` to comply to the spec (https://github.com/monitoring-mixins/docs/blob/master/design.pdf) BREAKING `prometheus_rules` is not available anymore

…fix gem installation (#682)

Adds an entirely new Dockerfile to the repository root which is capable of: - building promtail and loki from the same image - based on `scratch`, so that the final image can be executed on every supported GOARCH Please note that this Dockerfile should always be built using BuildKit (`buildctl` or `DOCKER_BUILDKIT=1` in recent versions) for maximum performance, as BuildKit's DAG allows for smart skipping of uneeded stages. These changes were proposed in #659

* fix(loki): pass missing config error to user Missing config errors are handled at the library level. Our own check mitigated this and causes loki to SEGFAULT later on * feat(loki): default config file in container The container provides a default config file. Use it by default * Revert "fix(loki): pass missing config error to user" This reverts commit b2744fc, because loki it assumed loki was incapable of running without config, which is not the case.

* move Dockerfile multi-arch to build and ignore that folder * docker-driver fix

* Query label values and names are now fetched from the store. A time range is now required by the /api/prom/label with a sane default (6 hours from now). * fix http querystring and update doc * update vendor * rebased

canary will suspend all operations on SIGINT but not exit, allowing you to shutdown the canary without it being restarted by docker/kubernetes SIGTERM will shutdown everything and end the process

…nd nanosecond timestamps

…to strings, this way we can eliminate non significant trailing zeros such that the float value 1 would be "1" as a string instead of "1.000000"

* feat(logcli): query from absolute timestamp Adds a new flag (`--from`) as to complement `--since`. While since subtracts a relative duration from the current time, from allows to specify the absolute start of the lookback window. Note: `--from` takes precedence over `--since`, but only if set. * fix(logcli): use RFC3339Nano for -from To comply with Prometheus, this flag now honors Nanoseconds as well (was using RFC3339 so far)

* feat(logcli): quiet mode Adds a quiet mode (-q / --quiet) to suppress the debug messages to stderr * feat(logcli): output modes Adds two alternative output modes (-o / --output) - raw: emits the line as parsed - jsonl: emits the line plus all known metadata as JSONL (JSON Line) Usage: -o [default, raw, jsonl] * feat(logcli): quiet tailing mode * feat(logcli): output modes while tailing Supports the three different output modes in tail mode as well * feat(logcli): print labels in jsonl tail mode * refactor(logcli): clean up entry printing Moves the entry printing into a standardized interface, implements this three times: - default (human readable) - jsonl (for scripts) - raw ('as is')

Reduces background chatter which in turn can trigger more log writes Signed-off-by: Brian Candler <b.candler@pobox.com>

…s unset (#750) Fixes #749 Signed-off-by: Brian Candler <b.candler@pobox.com>

Support for reading systemd journal entries has been added. promtail will look for a job in scrape_configs with a journal key to activate the journal target. If GOOS=linux and CGO_ENABLED=1, promtail will now require libsystemd headers to be available for building. If GOOS is not linux or CGO_ENABLED is not 1, journal support will be unavailable and a log message will be printed warning the user that their config file has journal tailing configured without it being built into promtail. See docs/promtail-examples.md for a concrete example of using journal support. Other structural changes made: 1. Ability for checking if scrape.Config.ServiceDiscoveryConfig is non-zero has been added. This was chosen over making ServiceDiscoveryConfig a pointer as yaml.v2 cannot parse an inline struct into a pointer value. 2. Updated pipeline logger component name to journal_pipeline and file_pipeline for JournalTargetManager and FileTargetManager respectively. 3. The positions file will now store positions as strings instead of integers. Existing positions will be read in properly but written out as strings the next time the positions file is saved. This is done to be able to store the journal cursor, which is a string. The positions API has been updated to support reading in the old integer values and the new string values.

cleanup rake warnings, bump version

This commit rolls back the changes to the Makefile and build/Dockerfile that caused CGO_ENABLED=1 to be present during some builds. This commit causes the journal support to be disabled in any build produced by make. Journal support can still be enabled in a manual build: go build -o cmd/promtail/promtail cmd/promtail

* add benchmark for storage queries * improve iterator to load only on next * fix memory retained by lazy chunks * reverse backward lazy iterator

…1 hour (#756) * Added tail length limit to limit duration of live tailing of logs to 1 hour * Some code refactoring suggested from PR review for live tailing duration limit * Fixed error messages in live tailing of logs * Fixed lint errors

moving loki-canary project under loki repo

* rounds nanoseconds boundaries to milliseconds * convert also store query

…776) Fixes #774

…the TLSConfig (#778)

* refactor(querier/ingester): TailRequest Lookback window Moves the specifications of the Lookback Window out of the logproto.QueryRequest into it's own type logproto.Lookback. This is required, because the Lookback Window will be used in the TailRequest as well. * feat(querier): parse Lookback from HTTP Request * feat(logcli): send Lookback Window spec with tail request * feat(querier): include historic entries in tail mode Extends tailing by sending a configurable amount of historic entries with before the live entries. This enables a behaviour that is closer to kubectl logs -f and docker logs -f. It is implemented by running a regular Query before subscribing to the ingesters. * fix: adapt tests to Lookback change * feat(querier): check all errors to make the linter happy * fix(logproto): flatten Lookback window spec Flattens the Lookback window spec into the individual queries * fix(ingester): adapt test to Lookback flatten

* Use prometheus pool for line buffer.

- document centos dep - add `-y` option Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

Document centos dep

This commit introduces chunk transfers, borrowing the mechanism from Cortex's implementation: when an ingester is shut down with claim on rollout enabled, the ingester will find a pending ingester and transfer all of its chunks to it.

…show you the name of the pipeline stage which just processed the log remove some unnecessary logging around fsnotify events we don't care about and saving positions Make the Processing Log Lines doc a first class citizen, I reference this a lot and currently it's hidden behind 3 clicks

ingester: support chunk transfers on ingester shutdown.

PR #791 accidentally removed the ability to use the promtail UI by removing methods from the Target interface that were only used within the HTML templates. Along with restoring the methods in the Target interface, this commit also introduces details for the JournalTarget, which currently only provides the position in the journal being tracked.

Registering the Ingester service to gRPC enables the chunk transfer mechanism to work.

refactor(makefile): simplify makefile by removing the autogenerated targets and make building binaries and dockerfiles separate operations.

… to make called with BUILD_IN_CONTAINER=false from circleci

* fix tail library logs to use our own log format * PR Feedbacks

Commits on Jun 14, 2019

rename fluent plugin and update docs

briangann committed Jun 14, 2019

Configuration menu

View commit details

Copy full SHA for ff2c567

Browse repository at this point

Copy the full SHA

ff2c567 View commit details

Browse the repository at this point in the history

Commits on Jun 24, 2019

adding resource requests in jsonnet

slim-bean committed Jun 24, 2019

Configuration menu

View commit details

Copy full SHA for 4bfcd09

Browse repository at this point

Copy the full SHA

4bfcd09 View commit details

Browse the repository at this point in the history

Commits on Jul 24, 2019

ingester: clean up chunk transfer code

rfratto committed Jul 24, 2019

Configuration menu

View commit details

Copy full SHA for 15cbba7

Browse repository at this point

Copy the full SHA

15cbba7 View commit details

Browse the repository at this point in the history

latest master #1

latest master #1

Commits on Apr 26, 2019

Commits on Apr 28, 2019

Commits on Apr 29, 2019

Commits on Apr 30, 2019

Commits on May 1, 2019

Commits on May 2, 2019

Commits on May 3, 2019

Commits on May 5, 2019

Commits on May 6, 2019

Commits on May 7, 2019

Commits on May 8, 2019

Commits on May 9, 2019

Commits on May 10, 2019

Commits on May 14, 2019

Commits on May 15, 2019

Commits on May 16, 2019

Commits on May 17, 2019

Commits on May 22, 2019

Commits on May 23, 2019

Commits on May 24, 2019

Commits on May 27, 2019

Commits on May 29, 2019

Commits on May 30, 2019

Commits on May 31, 2019

Commits on Jun 1, 2019

Commits on Jun 3, 2019

Commits on Jun 4, 2019

Commits on Jun 6, 2019

Commits on Jun 7, 2019

Commits on Jun 10, 2019

Commits on Jun 12, 2019

Commits on Jun 13, 2019

Commits on Jun 14, 2019

Commits on Jun 17, 2019

Commits on Jun 18, 2019

Commits on Jun 19, 2019

Commits on Jun 20, 2019

Commits on Jun 24, 2019

Commits on Jun 26, 2019

Commits on Jul 1, 2019

Commits on Jul 2, 2019

Commits on Jul 5, 2019

Commits on Jul 8, 2019

Commits on Jul 9, 2019

Commits on Jul 10, 2019

Commits on Jul 11, 2019

Commits on Jul 15, 2019

Commits on Jul 16, 2019

Commits on Jul 17, 2019

Commits on Jul 18, 2019

Commits on Jul 19, 2019

Commits on Jul 22, 2019

Commits on Jul 23, 2019

Commits on Jul 24, 2019

Commits on Jul 25, 2019

Commits on Jul 26, 2019

Commits on Jul 29, 2019

Commits on Jul 30, 2019