Update to v1.30 (#333)

* Update envoy dependency to v1.30.1

* Change repository back to upstream envoy

* Remove wavm

This was removed from upstream - see envoyproxy/envoy#32872

* Add some compile flags to get CEL to compile

These flags are copied from upstream's .bazelrc. The most important one here is
the `-fsized-deallocation` which was required to get CEL to compile. The rest
may not be necessary, but since they changed in upstream from 1.29 to 1.30, I
think it might be a good idea to include them.

* Get matchers working

There were some details in regex that were changed upstream that we had to
incorporate

* Fix a few flags that were moved

* Get HTTP transformation to compile

A few upstream Envoy functions that were being called in this library needed to
be passed contexts, which required plumbing that object through the call
hierarchy. Along the way, it seemed to appear that there were a few type
signatures that were possibly not completely correct, and those have been
adjusted to get things compiling successfully.

* Fix some compile errors

It seems that passing string arguments constructed with the `+` operator is not
allowed in ENVOY_STREAM_LOG, and possibly other macros as well. This was my
first encounter with this error, but there might be other instances lurking
elsewhere in the repository.

* addWatch callback must now return absl::Status

* Fix type signature of createProtocolOptionsConfig

The signature of this abstract class method was changed upstream - this change
reflects the fix

* Add changelog

* Update extensions_build_config.bzl

* Move changelog

* Get inja_transformer_test.cc to compile

Interesting lesson here - when calling `fmt::format`, we must use a constexpr,
since the formatting function provides compile-time type checking. Neat!

* Fix a broken test

* Add router_check tool

Upstream ci calls this, so we put in a dummy target here

* Update bazel version

* do_ci.sh debug flags

* Update envoy-build-ubuntu image

* Remove debug lines from do_ci.sh

* Remove envoy.string_matcher.lua

Co-authored-by: Nathan Fudenberg <nathan.fudenberg@solo.io>

* Remove envoy.tracers.opentelemetry.samplers.dynatrace from bazel/extensions/extensions_build_config.bzl

Co-authored-by: Nathan Fudenberg <nathan.fudenberg@solo.io>

* Update changelog/v1.30.1-patch1/update-to-upstream-envoy-v1.30.yaml

Co-authored-by: Nathan Fudenberg <nathan.fudenberg@solo.io>

* Update source/extensions/filters/http/aws_lambda/config.cc

Co-authored-by: Nathan Fudenberg <nathan.fudenberg@solo.io>

* Revert "Update source/extensions/filters/http/aws_lambda/config.cc"

This reverts commit f91e837.

* Fix one more review comment

* Revert `fmt::vformat` change in inja_transformer_test.cc

* Add comments explaining why `//test/tools` exist

* CHange auto variables to `std::string`

* Change `fmt::vformat` back to `fmt::format`

Previous iterations were still using the `vformat` call so they may not have
been using the `constexpr` versino of formatting correctly

---------

Co-authored-by: Nathan Fudenberg <nathan.fudenberg@solo.io>

.bazelrc

@@ -55,9 +55,11 @@ common --experimental_allow_tags_propagation
 
 # Enable position independent code (this is the default on macOS and Windows)
 # (Workaround for https://github.com/bazelbuild/rules_foreign_cc/issues/421)
+build:linux --copt=-fdebug-types-section
 build:linux --copt=-fPIC
 build:linux --copt=-Wno-deprecated-declarations
-build:linux --cxxopt=-std=c++17 --host_cxxopt=-std=c++17
+build:linux --cxxopt=-std=c++20 --host_cxxopt=-std=c++20
+build:linux --cxxopt=-fsized-deallocation --host_cxxopt=-fsized-deallocation # needed to compile CEL on Envoy 1.30.x and above
 build:linux --conlyopt=-fexceptions
 build:linux --fission=dbg,opt
 build:linux --features=per_object_debug_info
@@ -117,7 +119,7 @@ build:clang-asan --linkopt --rtlib=compiler-rt
 build:clang-asan --linkopt --unwindlib=libgcc
 
 # macOS
-build:macos --cxxopt=-std=c++17 --host_cxxopt=-std=c++17
+build:macos --cxxopt=-std=c++20 --host_cxxopt=-std=c++20
 build:macos --action_env=PATH=/opt/homebrew/bin:/opt/local/bin:/usr/local/bin:/usr/bin:/bin
 build:macos --host_action_env=PATH=/opt/homebrew/bin:/opt/local/bin:/usr/local/bin:/usr/bin:/bin
 build:macos --define tcmalloc=disabled

.bazelversion

@@ -1 +1 @@
-6.3.2
+6.5.0

bazel/extensions/extensions_build_config.bzl

@@ -7,6 +7,7 @@ EXTENSIONS = {
 
     "envoy.access_loggers.file":                        "//source/extensions/access_loggers/file:config",
     "envoy.access_loggers.extension_filters.cel":       "//source/extensions/access_loggers/filters/cel:config",
+    "envoy.access_loggers.fluentd"  :                   "//source/extensions/access_loggers/fluentd:config",
     "envoy.access_loggers.http_grpc":                   "//source/extensions/access_loggers/grpc:http_config",
     "envoy.access_loggers.tcp_grpc":                    "//source/extensions/access_loggers/grpc:tcp_config",
     "envoy.access_loggers.open_telemetry":              "//source/extensions/access_loggers/open_telemetry:config",
@@ -114,6 +115,11 @@ EXTENSIONS = {
 
     # "envoy.matching.actions.format_string":             "//source/extensions/matching/actions/format_string:config",
 
+    #
+    # StringMatchers
+    #
+#    "envoy.string_matcher.lua":                         "//source/extensions/string_matcher/lua:config",
+
     #
     # HTTP filters
     #
@@ -132,6 +138,7 @@ EXTENSIONS = {
     "envoy.filters.http.cors":                          "//source/extensions/filters/http/cors:config",
     "envoy.filters.http.composite":                     "//source/extensions/filters/http/composite:config",
     # "envoy.filters.http.connect_grpc_bridge":           "//source/extensions/filters/http/connect_grpc_bridge:config",
+    "envoy.filters.http.credential_injector":           "//source/extensions/filters/http/credential_injector:config",
     "envoy.filters.http.csrf":                          "//source/extensions/filters/http/csrf:config",
     # "envoy.filters.http.custom_response":               "//source/extensions/filters/http/custom_response:factory",
     "envoy.filters.http.decompressor":                  "//source/extensions/filters/http/decompressor:config",
@@ -276,6 +283,7 @@ EXTENSIONS = {
     #
 
     "envoy.tracers.opentelemetry.samplers.always_on":         "//source/extensions/tracers/opentelemetry/samplers/always_on:config",
+#    "envoy.tracers.opentelemetry.samplers.dynatrace":         "//source/extensions/tracers/opentelemetry/samplers/dynatrace:config",
 
     #
     # Transport sockets
@@ -288,6 +296,7 @@ EXTENSIONS = {
     "envoy.transport_sockets.tap":                      "//source/extensions/transport_sockets/tap:config",
     "envoy.transport_sockets.starttls":                 "//source/extensions/transport_sockets/starttls:config",
     "envoy.transport_sockets.tcp_stats":                "//source/extensions/transport_sockets/tcp_stats:config",
+    "envoy.transport_sockets.tls":                      "//source/extensions/transport_sockets/tls:config",
     "envoy.transport_sockets.internal_upstream":        "//source/extensions/transport_sockets/internal_upstream:config",
 
     #
@@ -339,7 +348,6 @@ EXTENSIONS = {
     "envoy.wasm.runtime.null":                          "//source/extensions/wasm_runtime/null:config",
     "envoy.wasm.runtime.v8":                            "//source/extensions/wasm_runtime/v8:config",
     "envoy.wasm.runtime.wamr":                          "//source/extensions/wasm_runtime/wamr:config",
-    "envoy.wasm.runtime.wavm":                          "//source/extensions/wasm_runtime/wavm:config",
     "envoy.wasm.runtime.wasmtime":                      "//source/extensions/wasm_runtime/wasmtime:config",
 
     #
@@ -388,6 +396,12 @@ EXTENSIONS = {
     # "envoy.http.custom_response.redirect_policy":             "//source/extensions/http/custom_response/redirect_policy:redirect_policy_lib",
     # "envoy.http.custom_response.local_response_policy":       "//source/extensions/http/custom_response/local_response_policy:local_response_policy_lib",
 
+    #
+    # Injected credentials
+    #
+
+    "envoy.http.injected_credentials.generic":              "//source/extensions/http/injected_credentials/generic:config",
+
     #
     # QUIC extensions
     #

bazel/repository_locations.bzl

@@ -2,9 +2,9 @@ REPOSITORY_LOCATIONS = dict(
     # can't have more than one comment between envoy line and commit line in 
     # order to accommodate `check_extensions_build_config.sh`
     envoy = dict(
-        # envoy 1.29.3 with backported ext_proc updates
-        commit = "e2dab93e60e93b56fa63632fb9cfa64930fc5240", # v1.29.3-fork1
-        remote = "https://github.com/solo-io/envoy-fork",
+        # envoy v1.30.1
+        commit = "816188b86a0a52095b116b107f576324082c7c02",
+        remote = "https://github.com/envoyproxy/envoy",
     ),
     inja = dict(
         # Includes unmerged modifications for

changelog/v1.30.1-patch1/update-to-upstream-envoy-v1.30.yaml

@@ -0,0 +1,10 @@
+changelog:
+- type: DEPENDENCY_BUMP
+  dependencyOwner: envoyproxy
+  dependencyRepo: envoy
+  dependencyTag: v1.30.1
+  resolvesIssue: false
+  description: >-
+    Update Envoy to v1.30.1 and switch back off our fork
+  issueLink: https://github.com/solo-io/envoy-gloo-ee/issues/768 
+

ci/cloudbuild.yaml

@@ -1,5 +1,5 @@
 steps:
-- name: 'envoyproxy/envoy-build-ubuntu:41c5a05d708972d703661b702a63ef5060125c33'
+- name: 'envoyproxy/envoy-build-ubuntu:f94a38f62220a2b017878b790b6ea98a0f6c5f9c'
   id: 'do_ci'
   args: ['ci/do_ci.sh', 'release', '//test/extensions/...', '//test/common/...', '//test/integration/...']
   volumes:

source/common/matcher/BUILD

@@ -19,8 +19,10 @@ envoy_cc_library(
     deps = [
         "@envoy//source/common/router:config_lib",
         "@envoy//source/common/http:header_utility_lib",
+        "@envoy//envoy/common:regex_interface",
         "@envoy_api//envoy/api/v2/route:pkg_cc_proto",
         "@envoy_api//envoy/config/route/v3:pkg_cc_proto",
+        "//source/common/regex:regex_lib",
     ],
 )
 

source/common/matcher/solo_matcher.cc

@@ -1,7 +1,9 @@
 #include "source/common/matcher/solo_matcher.h"
 
+#include "envoy/common/regex.h"
 #include "source/common/common/logger.h"
 #include "source/common/common/regex.h"
+#include "source/common/regex/regex.h"
 #include "source/common/router/config_impl.h"
 
 #include "absl/strings/match.h"
@@ -19,15 +21,16 @@ namespace {
 class BaseMatcherImpl : public Matcher,
                         public Logger::Loggable<Logger::Id::filter> {
 public:
-  BaseMatcherImpl(const RouteMatch &match)
+  BaseMatcherImpl(const RouteMatch &match,
+                  Server::Configuration::CommonFactoryContext &context)
       : case_sensitive_(
             PROTOBUF_GET_WRAPPED_OR_DEFAULT(match, case_sensitive, true)),
-        config_headers_(
-            Http::HeaderUtility::buildHeaderDataVector(match.headers())) {
+        config_headers_(Http::HeaderUtility::buildHeaderDataVector(
+            match.headers(), context)) {
     for (const auto &query_parameter : match.query_parameters()) {
       config_query_parameters_.push_back(
           std::make_unique<Router::ConfigUtility::QueryParameterMatcher>(
-              query_parameter));
+              query_parameter, context));
     }
   }
 
@@ -38,9 +41,8 @@ class BaseMatcherImpl : public Matcher,
 
     matches &= Http::HeaderUtility::matchHeaders(headers, config_headers_);
     if (!config_query_parameters_.empty()) {
-      auto query_parameters =
-          Http::Utility::QueryParamsMulti::parseQueryString(
-              headers.Path()->value().getStringView());
+      auto query_parameters = Http::Utility::QueryParamsMulti::parseQueryString(
+          headers.Path()->value().getStringView());
       matches &= ConfigUtility::matchQueryParams(query_parameters,
                                                  config_query_parameters_);
     }
@@ -61,8 +63,9 @@ class BaseMatcherImpl : public Matcher,
  */
 class PrefixMatcherImpl : public BaseMatcherImpl {
 public:
-  PrefixMatcherImpl(const ::RouteMatch &match)
-      : BaseMatcherImpl(match), prefix_(match.prefix()) {}
+  PrefixMatcherImpl(const ::RouteMatch &match,
+                    Server::Configuration::CommonFactoryContext &context)
+      : BaseMatcherImpl(match, context), prefix_(match.prefix()) {}
 
   bool matches(const Http::RequestHeaderMap &headers) const override {
     if (BaseMatcherImpl::matchRoute(headers) &&
@@ -87,8 +90,9 @@ class PrefixMatcherImpl : public BaseMatcherImpl {
  */
 class PathMatcherImpl : public BaseMatcherImpl {
 public:
-  PathMatcherImpl(const ::RouteMatch &match)
-      : BaseMatcherImpl(match), path_(match.path()) {}
+  PathMatcherImpl(const ::RouteMatch &match,
+                  Server::Configuration::CommonFactoryContext &context)
+      : BaseMatcherImpl(match, context), path_(match.path()) {}
 
   bool matches(const Http::RequestHeaderMap &headers) const override {
     if (BaseMatcherImpl::matchRoute(headers)) {
@@ -112,15 +116,54 @@ class PathMatcherImpl : public BaseMatcherImpl {
   const std::string path_;
 };
 
+class CompiledStdMatcher : public Regex::CompiledMatcher {
+public:
+  CompiledStdMatcher(std::regex &&regex) : regex_(std::move(regex)) {}
+
+  // CompiledMatcher
+  bool match(absl::string_view value) const override {
+    try {
+      return std::regex_match(value.begin(), value.end(), regex_);
+    } catch (const std::regex_error &e) {
+      return false;
+    }
+  }
+
+  // CompiledMatcher
+  std::string replaceAll(absl::string_view value,
+                         absl::string_view substitution) const override {
+    try {
+      return std::regex_replace(std::string(value), regex_,
+                                std::string(substitution));
+    } catch (const std::regex_error &e) {
+      return std::string(value);
+    }
+  }
+
+private:
+  const std::regex regex_;
+};
+
+class StdRegexEngine : public Regex::Engine {
+public:
+  Regex::CompiledMatcherPtr matcher(const std::string &regex) const override {
+    return std::make_unique<CompiledStdMatcher>(
+        Solo::Regex::Utility::parseStdRegex(regex));
+  }
+};
+
 /**
  * Perform a match against any path with a regex rule.
  * TODO(mattklein123): This code needs dedup with RegexRouteEntryImpl.
  */
 class RegexMatcherImpl : public BaseMatcherImpl {
 public:
-  RegexMatcherImpl(const RouteMatch &match) : BaseMatcherImpl(match) {
+  RegexMatcherImpl(const RouteMatch &match,
+                   Server::Configuration::CommonFactoryContext &context)
+      : BaseMatcherImpl(match, context) {
     ASSERT(match.path_specifier_case() == RouteMatch::kSafeRegex);
-    regex_ = Regex::Utility::parseRegex(match.safe_regex());
+    auto engine = Envoy::MatcherCopy::StdRegexEngine();
+    regex_ = Regex::Utility::parseRegex(match.safe_regex(), engine);
     regex_str_ = match.safe_regex().regex();
   }
 
@@ -140,56 +183,29 @@ class RegexMatcherImpl : public BaseMatcherImpl {
   }
 
 private:
-
-static Regex::CompiledMatcherPtr parseStdRegexAsCompiledMatcher(const std::string& regex,
-                                 std::regex::flag_type flags = std::regex::optimize);
   Regex::CompiledMatcherPtr regex_;
   // raw regex string, for logging.
   std::string regex_str_;
 };
 
-class CompiledStdMatcher : public Regex::CompiledMatcher {
-public:
-  CompiledStdMatcher(std::regex&& regex) : regex_(std::move(regex)) {}
-
-  // CompiledMatcher
-  bool match(absl::string_view value) const override {
-    try {
-      return std::regex_match(value.begin(), value.end(), regex_);
-    } catch (const std::regex_error& e) {
-      return false;
-    }
-  }
-
-  // CompiledMatcher
-  std::string replaceAll(absl::string_view value, absl::string_view substitution) const override {
-    try {
-      return std::regex_replace(std::string(value), regex_, std::string(substitution));
-    } catch (const std::regex_error& e) {
-      return std::string(value);
-    }
-  }
-
-private:
-  const std::regex regex_;
-};
-
 } // namespace
 
-MatcherConstPtr Matcher::create(const RouteMatch &match) {
+MatcherConstPtr
+Matcher::create(const RouteMatch &match,
+                Server::Configuration::CommonFactoryContext &context) {
   switch (match.path_specifier_case()) {
   case RouteMatch::PathSpecifierCase::kPrefix:
-    return std::make_shared<PrefixMatcherImpl>(match);
+    return std::make_shared<PrefixMatcherImpl>(match, context);
   case RouteMatch::PathSpecifierCase::kPath:
-    return std::make_shared<PathMatcherImpl>(match);
+    return std::make_shared<PathMatcherImpl>(match, context);
   case RouteMatch::PathSpecifierCase::kSafeRegex:
-    return std::make_shared<RegexMatcherImpl>(match);
+    return std::make_shared<RegexMatcherImpl>(match, context);
   // path specifier is required.
   case RouteMatch::PathSpecifierCase::PATH_SPECIFIER_NOT_SET:
   default:
     PANIC_DUE_TO_CORRUPT_ENUM;
   }
 }
 
-} // namespace Matcher
+} // namespace MatcherCopy
 } // namespace Envoy

source/common/matcher/solo_matcher.h

@@ -2,6 +2,7 @@
 
 #include "envoy/config/route/v3/route.pb.h"
 #include "envoy/http/header_map.h"
+#include "envoy/server/factory_context.h"
 
 namespace Envoy {
 namespace MatcherCopy {
@@ -34,7 +35,8 @@ class Matcher {
    */
 
   static MatcherConstPtr
-  create(const ::envoy::config::route::v3::RouteMatch &match);
+  create(const ::envoy::config::route::v3::RouteMatch &match,
+         Server::Configuration::CommonFactoryContext& context);
 };
 
 } // namespace Matcher

source/extensions/filters/http/aws_lambda/aws_lambda_filter_config_factory.cc

@@ -41,7 +41,7 @@ AWSLambdaFilterConfigFactory::createFilterFactoryFromProtoTyped(
       };
 }
 
-Upstream::ProtocolOptionsConfigConstSharedPtr
+absl::StatusOr<Upstream::ProtocolOptionsConfigConstSharedPtr>
 AWSLambdaFilterConfigFactory::createProtocolOptionsConfig(
     const Protobuf::Message &config,
     Server::Configuration::ProtocolOptionsFactoryContext &) {

source/extensions/filters/http/aws_lambda/aws_lambda_filter_config_factory.h

@@ -23,7 +23,7 @@ class AWSLambdaFilterConfigFactory
   AWSLambdaFilterConfigFactory()
       : FactoryBase(SoloHttpFilterNames::get().AwsLambda) {}
 
-  Upstream::ProtocolOptionsConfigConstSharedPtr createProtocolOptionsConfig(
+  absl::StatusOr<Upstream::ProtocolOptionsConfigConstSharedPtr> createProtocolOptionsConfig(
       const Protobuf::Message &config,
       Server::Configuration::ProtocolOptionsFactoryContext &) override;
   ProtobufTypes::MessagePtr createEmptyProtocolOptionsProto() override;

source/extensions/filters/http/aws_lambda/config.cc

@@ -194,12 +194,18 @@ void AWSLambdaConfigImpl::AWSLambdaStsRefresher::init(Event::Dispatcher &dispatc
     } else {
         ENVOY_LOG(debug, "{}: STS enabled without time based refresh",__func__);
     }
-    file_watcher_->addWatch(
+    absl::Status result = file_watcher_->addWatch(
         parent_->token_file_, Filesystem::Watcher::Events::Modified,
         [shared_this](uint32_t) {
           // Force timer callback to happen immediately to pick up the change.
           shared_this->timer_->enableTimer(std::chrono::milliseconds::zero());
+          return absl::OkStatus();
         });
+    // the result of the above function MUST NOT be ignored (see [[nodiscard]]).
+    // since we don't expect an error to occur, we'll just log a message for now
+    if (!result.ok()) {
+      ENVOY_LOG(error, "Unexpected error occurred on file watcher timer: {}", result.message());
+    }
 }
 
 /*

source/extensions/filters/http/nats/streaming/nats_streaming_filter.cc

@@ -116,12 +116,12 @@ void NatsStreamingFilter::onResponse() { onCompletion(Http::Code::OK, ""); }
 
 void NatsStreamingFilter::onFailure() {
   onCompletion(Http::Code::InternalServerError, "nats streaming filter abort",
-               StreamInfo::ResponseFlag::NoHealthyUpstream);
+               StreamInfo::CoreResponseFlag::NoHealthyUpstream);
 }
 
 void NatsStreamingFilter::onTimeout() {
   onCompletion(Http::Code::RequestTimeout, "nats streaming filter timeout",
-               StreamInfo::ResponseFlag::UpstreamRequestTimeout);
+               StreamInfo::CoreResponseFlag::UpstreamRequestTimeout);
 }
 
 void NatsStreamingFilter::retrieveRouteSpecificFilterConfig() {