Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Backport of [HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration into release/1.16.x #18350

Closed
wants to merge 268 commits into from
Closed

Backport of [HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration into release/1.16.x #18350

wants to merge 268 commits into from

Conversation

hc-github-team-consul-core
Copy link
Contributor

Backport

This PR is auto-generated from #18168 to be assessed for backporting due to the inclusion of the label backport/1.16.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@Achooo
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

Description

New feature for HCP Telemetry that allows us to dynamically re-configure metrics export to HCP.

Background
Currently, we need to restart the Consul server to bring in any updates to filters, a new endpoint or new labels from the Consul Telemetry Gateway in HCP.

Ideally, we are able to modify these dynamically, which requires logic to continuously fetch new configuration and update it as necessary.

Implementation
This PR enables this functionality, with changes to :

  1. New object: TelemetryConfigProvider:
  • Periodically fetches telemetry configuration from HCP, using a time.Ticker.
  • Holds configuration and exposes it via methods : GetEndpoint(), GetFilters() and GetLabels()
  • Updates configuration, ONLY if it has changed. Comparisons are done using a hash function on the structs holding the config.
  • Handles concurrency access to read the config and modify the config.
  1. OTELSink: uses a new ConfigProvider interface to access filters and labels.
  • Since labels are now dynamic, we can't add them to the OTEL resource (immutable) so a new function is added to compute OTEL labels dynamically.
  • Handle dynamic filters
  1. OTELExporter: uses a new EndpointProvider interface to access the endpoint.

** Cleanup / Refactor ***

  1. The client package was cleaned to separate out the TelemetryConfig object to parse and convert objects.
  2. A logger was added to the client to log bad filter information

This refactor was necessary since now the TelemetryConfigProvider also fetches telemetry config. This avoid duplication of logic for the parsing of the endpoint, labels ,etc.

Testing & Reproduction steps

  • End to end tests:
  • Unit tests:

Links

PR Checklist

  • updated test coverage
  • external facing docs updated
  • appropriate backport labels added
  • not a security concern
Overview of commits

zalimeni and others added 30 commits June 12, 2023 14:28
@zalimeni
Post 1.16.0-rc1 updates (#17663)
446a640 
- Update changelog to include new entries from release
- Update submodule versions to latest published
@ramramhariram
Update service-defaults.mdx (#17656)
290ba0e
@boruszak @sfarkas-hashicorp @trujillo-adam
docs: Sameness Groups (#17628)
ef77f9a 
* port from enterprise branch

* Apply suggestions from code review

Co-authored-by: shanafarkas <105076572+shanafarkas@users.noreply.github.com>

* Update website/content/docs/connect/cluster-peering/usage/create-sameness-groups.mdx

* next steps

* Update website/content/docs/connect/cluster-peering/usage/create-sameness-groups.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/k8s/connect/cluster-peering/usage/create-sameness-groups.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

---------

Co-authored-by: shanafarkas <105076572+shanafarkas@users.noreply.github.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Remove "BETA" marker from config entries (#17670)
c384f24
@trujillo-adam @boruszak @Jeff-Apple
CAPIgw for K8s installation updates for 1.16 (#17627)
27206d9 
* trimmed CRD step and reqs from installation

* updated tech specs

* Apply suggestions from code review

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>

* added upgrade instruction

* removed tcp port req

* described downtime and DT-less upgrades

* applied additional review feedback

---------

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>
@trujillo-adam @Jeff-Apple
additional feedback on API gateway upgrades (#17677)
b678742 
* additional feedback

* Update website/content/docs/api-gateway/upgrades.mdx

Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>

---------

Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>
@boruszak @trujillo-adam
docs: JWT Authorization for intentions (#17643)
66704e5 
* Initial page/nav creation

* configuration entry reference page

* Usage + fixes

* service intentions page

* usage

* description

* config entry updates

* formatting fixes

* Update website/content/docs/connect/config-entries/service-intentions.mdx

Co-authored-by: Paul Glass <pglass@hashicorp.com>

* service intentions review fixes

* Overview page review fixes

* Apply suggestions from code review

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

---------

Co-authored-by: Paul Glass <pglass@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
@boruszak
docs: minor fixes to JWT auth docs (#17680)
37a13dc 
* Fixes

* service intentions fixes
Fix two WAL metrics in docs/agent/telemetry.mdx (#17593)
28d81ec
@trujillo-adam
updated failover for k8s w-tproxy page title (#17683)
0ddafcf
@im2nguyen @trujillo-adam
@boruszak @Jeff-Apple
Add release notes 1.16 rc (#17665)
3a8fc61 
* Merge pull request #5773 from hashicorp/docs/rate-limiting-from-ip-addresses-1.16

updated docs for rate limiting for IP addresses - 1.16

* Merge pull request #5609 from hashicorp/docs/enterprise-utilization-reporting

Add docs for enterprise utilization reporting

* Merge pull request #5734 from hashicorp/docs/envoy-ext-1.16

Docs/envoy ext 1.16

* Add release notes for 1.16-rc

* Add consul-e license utlization reporting

* Update with rc absolute links

* Update with rc absolute links

* fix typo

* Apply suggestions from code review

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update to use callout component

* address typo

* docs: FIPS 140-2 Compliance (#17668)

* Page + nav + formatting

* link fix

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* link fix

* Apply suggestions from code review

Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>

* Update website/content/docs/enterprise/fips.mdx

---------

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>

* fix apigw install values file

* fix typos in release notes

---------

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Jeff Apple <79924108+Jeff-Apple@users.noreply.github.com>
@im2nguyen
fix release notes links (#17687)
421e9d8
@trujillo-adam
adding redirects for tproxy and envoy extensions (#17688)
11764a4 
* adding redirects

* Apply suggestions from code review
@im2nguyen
Fix FIPS copy (#17691)
4b843ae 
* fix release notes links

* fix typos on fips docs
@absolutelightning
[NET-4107][Supportability] Log Level set to TRACE and duration set to…
d54d5fb 
… 5m for consul-debug (#17596)

* changed duration to 5 mins and log level to trace

* documentation update

* change log
@cthain
ENT merge of ext-authz extension updates (#17684)
a8f1350
@cthain
docs: Update default values for Envoy extension proxy types (#17676)
ddce431
@DanStough
fix: stop peering delete routine on leader loss (#17483)
bba5cd8
@erichaberkorn
Refactor disco chain prioritize by locality structs (#17696)
0a1efe7 
This includes prioritize by localities on disco chain targets rather than
resolvers, allowing different targets within the same partition to have
different policies.
@rboyer
agent: remove agent cache dependency from service mesh leaf certifica…
72f991d 
…te management (#17075)

* agent: remove agent cache dependency from service mesh leaf certificate management

This extracts the leaf cert management from within the agent cache.

This code was produced by the following process:

1. All tests in agent/cache, agent/cache-types, agent/auto-config,
   agent/consul/servercert were run at each stage.

    - The tests in agent matching .*Leaf were run at each stage.

    - The tests in agent/leafcert were run at each stage after they
      existed.

2. The former leaf cert Fetch implementation was extracted into a new
   package behind a "fake RPC" endpoint to make it look almost like all
   other cache type internals.

3. The old cache type was shimmed to use the fake RPC endpoint and
   generally cleaned up.

4. I selectively duplicated all of Get/Notify/NotifyCallback/Prepopulate
   from the agent/cache.Cache implementation over into the new package.
   This was renamed as leafcert.Manager.

    - Code that was irrelevant to the leaf cert type was deleted
      (inlining blocking=true, refresh=false)

5. Everything that used the leaf cert cache type (including proxycfg
   stuff) was shifted to use the leafcert.Manager instead.

6. agent/cache-types tests were moved and gently replumbed to execute
   as-is against a leafcert.Manager.

7. Inspired by some of the locking changes from derek's branch I split
   the fat lock into N+1 locks.

8. The waiter chan struct{} was eventually replaced with a
   singleflight.Group around cache updates, which was likely the biggest
   net structural change.

9. The awkward two layers or logic produced as a byproduct of marrying
   the agent cache management code with the leaf cert type code was
   slowly coalesced and flattened to remove confusion.

10. The .*Leaf tests from the agent package were copied and made to work
    directly against a leafcert.Manager to increase direct coverage.

I have done a best effort attempt to port the previous leaf-cert cache
type's tests over in spirit, as well as to take the e2e-ish tests in the
agent package with Leaf in the test name and copy those into the
agent/leafcert package to get more direct coverage, rather than coverage
tangled up in the agent logic.

There is no net-new test coverage, just coverage that was pushed around
from elsewhere.
@curtbushko
[core]: Pin github action workflows (#17695)
0c15748
@DanStough
docs: missing changelog for _5517 (#17706)
d497623
@trujillo-adam @im2nguyen
add enterprise notes for IP-based rate limits (#17711)
ab909b4 
* add enterprise notes for IP-based rate limits

* Apply suggestions from code review

Co-authored-by: Tu Nguyen <im2nguyen@users.noreply.github.com>
Co-authored-by: David Yu <dyu@hashicorp.com>

* added bolded 'Enterprise' in list items.

---------

Co-authored-by: Tu Nguyen <im2nguyen@users.noreply.github.com>
Co-authored-by: David Yu <dyu@hashicorp.com>
Update compatibility.mdx (#17713)
28647ef
Remove extraneous version info for Config entries (#17716)
9acbe76 
* Update terminating-gateway.mdx
* Update exported-services.mdx
* Update mesh.mdx
@tcraxs @trujillo-adam
fix: typo in link to section (#17527)
8d9f2eb 
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Bump Alpine to 3.18 (#17719)
212e090 
* Update Dockerfile

* Create 17719.txt
@trujillo-adam @jkirschner-hashicorp
NET-1825: New ACL token creation docs (#16465)
6a90c23 
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
@absolutelightning @analogue
[NET-3865] [Supportability] Additional Information in the output of '…
fa40654 
…consul operator raft list-peers' (#17582)

* init

* fix tests

* added -detailed in docs

* added change log

* fix doc

* checking for entry in map

* fix tests

* removed detailed flag

* removed detailed flag

* revert unwanted changes

* removed unwanted changes

* updated change log

* pr review comment changes

* pr comment changes single API instead of two

* fix change log

* fix tests

* fix tests

* fix test operator raft endpoint test

* Update .changelog/17582.txt

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* nits

* updated docs

---------

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>
@cthain
OSS merge: Update error handling login when applying extensions (#17740)
9289e68
@hc-github-team-consul-core hc-github-team-consul-core force-pushed the backport/cc-4960/hcp-telemetry-periodic-refresh/quietly-splendid-leech branch 2 times, most recently from 2e52fc2 to 78b2180 Compare August 1, 2023 21:22
@hc-github-team-consul-core hc-github-team-consul-core force-pushed the backport/cc-4960/hcp-telemetry-periodic-refresh/quietly-splendid-leech branch from 78b2180 to 2e52fc2 Compare August 1, 2023 21:22
github-team-consul-core-pr-approver
github-team-consul-core-pr-approver approved these changes Aug 1, 2023
View reviewed changes
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto approved Consul Bot automated PR

@github-actions github-actions bot added type/docs Documentation needs to be created/updated/clarified theme/api Relating to the HTTP API interface theme/acls ACL and token generation theme/cli Flags and documentation for the CLI interface theme/config Relating to Consul Agent configuration, including reloading theme/ui Anything related to the UI theme/connect Anything related to Consul Connect, Service Mesh, Side Car Proxies type/ci Relating to continuous integration (CI) tooling for testing or releases pr/dependencies PR specifically updates dependencies of project theme/envoy/xds Related to Envoy support theme/contributing Additions and enhancements to community contributing materials theme/consul-terraform-sync Relating to Consul Terraform Sync and Network Infrastructure Automation labels Aug 1, 2023
@Achooo
Copy link
Contributor

Achooo commented Aug 2, 2023

Closed in favour of manual backport PR #18359

@Achooo Achooo mentioned this pull request Aug 2, 2023
Merged
5 tasks
@Achooo Achooo closed this Aug 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-team-consul-core-pr-approver github-team-consul-core-pr-approver github-team-consul-core-pr-approver approved these changes

@Achooo Achooo Awaiting requested review from Achooo

Assignees

@Achooo Achooo

Labels
pr/dependencies PR specifically updates dependencies of project theme/acls ACL and token generation theme/api Relating to the HTTP API interface theme/cli Flags and documentation for the CLI interface theme/config Relating to Consul Agent configuration, including reloading theme/connect Anything related to Consul Connect, Service Mesh, Side Car Proxies theme/consul-terraform-sync Relating to Consul Terraform Sync and Network Infrastructure Automation theme/contributing Additions and enhancements to community contributing materials theme/envoy/xds Related to Envoy support theme/ui Anything related to the UI type/ci Relating to continuous integration (CI) tooling for testing or releases type/docs Documentation needs to be created/updated/clarified
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.