es: drop jobs-* indices #10857
Merged
es: drop jobs-* indices #10857
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We are currently ingesting Bazel events in two forms: In the `events-*` indices, each Bazel event is recorded as a separate ES object, with the corresponding job name as a field that can serve to aggregate all of the events for a given job. In the `jobs-*` indices, each job is ingested as a single (composite) ES object, with the individual events as elements in a list-type field. When I set up the cluster, I wasn't sure which one would be more useful, so I included both. We now have a bit more usage experience and it turns out the `events-*` form is the only one we use, so I think we should stop ingesting evrything twice and from now on create only the `events-*` ones. CHANGELOG_BEGIN CHANGELOG_END
garyverhaegen-da
requested review from
cocreature,
aherrmann-da and
stefanobaghino-da
|
There's no urgency on this — I'd prefer to wait for @aherrmann-da's stamp. |
cocreature
approved these changes
Sep 13, 2021
Andreas will not be available for a couple of weeks. |
sure but @garyverhaegen-da said there is no urgency and he prefers to wait |
Absolutely, I just wanted to point out that feedback from Andreas will need to wait for a couple of weeks. 🙂 |
No rush in merging this, feel free to wait for Andreas' opinion on this before merging. |
garyverhaegen-da
added a commit
that referenced
this pull request
Sep 24, 2021
Disks are currently at 75% fullness, so seems like a good idea to bump a bit. #10857 should reduce our needs, too, so this should last us a while. CHANGELOG_BEGIN CHANGELOG_END
Merged
garyverhaegen-da
added a commit
that referenced
this pull request
Sep 25, 2021
Disks are currently at 75% fullness, so seems like a good idea to bump a bit. #10857 should reduce our needs, too, so this should last us a while. CHANGELOG_BEGIN CHANGELOG_END
|
Deployed. All seems to be running fine. |
azure-pipelines bot
pushed a commit
that referenced
this pull request
Sep 29, 2021
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @stefanobaghino-da is in charge of this release. Commit log: ``` b4d0031 detect unsynchronized contract table and retry (#10617) 3d779cf [Mutable state cache] Fix initialization offset (#11024) 5458aa8 Switch on NonUnitStatements warning in daml-lf/transaction (#11048) 9641fd5 auth middleware: no print secret (#11050) a885f52 [DPP-417] [DPP-595] Add error code version switching mechanism (#11035) 12e0c72 fix blackduck logic (#11049) eb87b34 kvutils: Add the logging context for ledger state operations. (#11030) b7daa5f Address remaining dependabot alerts (#11045) 5e43f8c es: drop jobs-* indices (#10857) 03203b7 Define encoding/decoding for module imports (#11036) 57a1597 Setting timeoutToleranceMillis to 10 minutes to prevent flakiness (#11043) df59f3f Fix Navigator dependabot alerts (#11044) 6bf45a3 Upgrade Navigator to Webpack 5 (#11040) 80e217e [DPP-622] Add conformance tests that verifies TLSv1.0 and TLSv1 are disabled. (#10983) 626e1fb Small lf value.cids regression fix (found by canton unit tests) (#11032) a4629a4 KV: Ignore daml_lf_1.proto when checking for KV protobuf compatibility (#11021) ee9be65 kvutils: Add metadata to `Err` [KVL-1032] (#10992) 0d3ae6e interface methods: Haskell Typechecker (#11028) e36eb46 Resolve `set-value` to 4.0.1 and above (#11029) 4075624 interface methods: Haskell AST (#11018) 7c1fd50 Bump grunt-browserify (#11026) 5f3f582 Upgrade webpack-dev-server in Navigator (#11025) 91be1e1 Drop matchdep dependency from docs build (#11023) fe9aeff Increase es disk size (#11019) eac7963 LF: Refactor ProtoTest.scala (#11020) e79a30a For the client binding propagate the full original completion [KVL-1112] (#10879) 59ad995 fix buf check (#11014) abc3e66 Increase the tolarance for handover of control in HaCoordinatorSpec (#10997) 19b2bf4 LF: Cosmetic clean-up in the Speedy Compiler (#11015) cb0e41f LF: Add interface support to the Preprocessor (#11013) c33297c Remove `transactionNormalization` flag. (#11010) f5d2135 Check protobuf compatibility of `main` and PR commits w.r.t. previous stable release [KVL-1109] (#10950) bf8b75d interface methods: Add protobuf definitions. (#11005) 88e1430 Make LargeTransactionTest use ValueEnricher, so it can work with normalized transactions coming out of the engine. (#11003) a043926 rotate release duty after 1.17.0-snapshot.20210921.7889.0.1b473c2b (#10972) 35666ca Add MINIMAL pragma for Additive type class (#11001) 8de162b [DPP-586] Upgrade to netty 4.1.67.Final and netty-tcnative-boringssl-static 2.0.40.Final (#10956) 871d03b release 1.18.0-snapshot.20210922.7908.0.ced4a272 (#10998) 721575e [JSON-API] Postgres perf job (#10986) f2d9f07 Release RC2 for SDK 1.17.0 (#10996) ``` Changelog: ``` - [JSON API] Under rare conditions, a multi-template query backed by database could have an ACS portion that doesn't match its transaction stream, if updated concurrently. This conditions is now checked and accounted for. See `issue #10617 <https://github.com/digital-asset/daml/pull/10617>`__. - The OAuth2 Middleware now obfuscates its Client Secret when logging its config. - [Integration Kit] We have added ``loggingContext`` as an implicit parameter to more _kvutils_ trait methods. Implementors may need to do the same in their trait implementations. This can make it easier to log with the appropriate context. java-client-bindings - the original full completion is included in the `CompletionResponse` when available Daml on SQL, Integration Kit, Sandbox: Drop support for TLS 1.0 and 1.1 in Ledger API. ``` CHANGELOG_BEGIN CHANGELOG_END
stefanobaghino-da
pushed a commit
that referenced
this pull request
Sep 29, 2021
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @stefanobaghino-da is in charge of this release. Commit log: ``` b4d0031 detect unsynchronized contract table and retry (#10617) 3d779cf [Mutable state cache] Fix initialization offset (#11024) 5458aa8 Switch on NonUnitStatements warning in daml-lf/transaction (#11048) 9641fd5 auth middleware: no print secret (#11050) a885f52 [DPP-417] [DPP-595] Add error code version switching mechanism (#11035) 12e0c72 fix blackduck logic (#11049) eb87b34 kvutils: Add the logging context for ledger state operations. (#11030) b7daa5f Address remaining dependabot alerts (#11045) 5e43f8c es: drop jobs-* indices (#10857) 03203b7 Define encoding/decoding for module imports (#11036) 57a1597 Setting timeoutToleranceMillis to 10 minutes to prevent flakiness (#11043) df59f3f Fix Navigator dependabot alerts (#11044) 6bf45a3 Upgrade Navigator to Webpack 5 (#11040) 80e217e [DPP-622] Add conformance tests that verifies TLSv1.0 and TLSv1 are disabled. (#10983) 626e1fb Small lf value.cids regression fix (found by canton unit tests) (#11032) a4629a4 KV: Ignore daml_lf_1.proto when checking for KV protobuf compatibility (#11021) ee9be65 kvutils: Add metadata to `Err` [KVL-1032] (#10992) 0d3ae6e interface methods: Haskell Typechecker (#11028) e36eb46 Resolve `set-value` to 4.0.1 and above (#11029) 4075624 interface methods: Haskell AST (#11018) 7c1fd50 Bump grunt-browserify (#11026) 5f3f582 Upgrade webpack-dev-server in Navigator (#11025) 91be1e1 Drop matchdep dependency from docs build (#11023) fe9aeff Increase es disk size (#11019) eac7963 LF: Refactor ProtoTest.scala (#11020) e79a30a For the client binding propagate the full original completion [KVL-1112] (#10879) 59ad995 fix buf check (#11014) abc3e66 Increase the tolarance for handover of control in HaCoordinatorSpec (#10997) 19b2bf4 LF: Cosmetic clean-up in the Speedy Compiler (#11015) cb0e41f LF: Add interface support to the Preprocessor (#11013) c33297c Remove `transactionNormalization` flag. (#11010) f5d2135 Check protobuf compatibility of `main` and PR commits w.r.t. previous stable release [KVL-1109] (#10950) bf8b75d interface methods: Add protobuf definitions. (#11005) 88e1430 Make LargeTransactionTest use ValueEnricher, so it can work with normalized transactions coming out of the engine. (#11003) a043926 rotate release duty after 1.17.0-snapshot.20210921.7889.0.1b473c2b (#10972) 35666ca Add MINIMAL pragma for Additive type class (#11001) 8de162b [DPP-586] Upgrade to netty 4.1.67.Final and netty-tcnative-boringssl-static 2.0.40.Final (#10956) 871d03b release 1.18.0-snapshot.20210922.7908.0.ced4a272 (#10998) 721575e [JSON-API] Postgres perf job (#10986) f2d9f07 Release RC2 for SDK 1.17.0 (#10996) ``` Changelog: ``` - [JSON API] Under rare conditions, a multi-template query backed by database could have an ACS portion that doesn't match its transaction stream, if updated concurrently. This conditions is now checked and accounted for. See `issue #10617 <https://github.com/digital-asset/daml/pull/10617>`__. - The OAuth2 Middleware now obfuscates its Client Secret when logging its config. - [Integration Kit] We have added ``loggingContext`` as an implicit parameter to more _kvutils_ trait methods. Implementors may need to do the same in their trait implementations. This can make it easier to log with the appropriate context. java-client-bindings - the original full completion is included in the `CompletionResponse` when available Daml on SQL, Integration Kit, Sandbox: Drop support for TLS 1.0 and 1.1 in Ledger API. ``` CHANGELOG_BEGIN CHANGELOG_END Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are currently ingesting Bazel events in two forms:
In the
events-*indices, each Bazel event is recorded as a separate ES object, with the corresponding job name as a field that can serve to aggregate all of the events for a given job.In the
jobs-*indices, each job is ingested as a single (composite) ES object, with the individual events as elements in a list-type field.When I set up the cluster, I wasn't sure which one would be more useful, so I included both. We now have a bit more usage experience and it turns out the
events-*form is the only one we use, so I think we should stop ingesting everything twice and from now on create only theevents-*ones.CHANGELOG_BEGIN
CHANGELOG_END