A simple filter that probably (hopefully) isn't of much production value, but demonstrates how to write a third-party Envoy filter that is compatible with both the v1 and v2 API. I hope that this repository serves as a nice guide for people who maybe don't want to dive into the internals of Envoy too deeply, but want to be able to build custom Envoy filters outside of the main source code.
I wanted to write this because the current defacto example doesn't use any configuration data, which is one of the parts I wrestled with the most in building this filter. As a young whippersnapper new-grad, I thought that this "outside perspective" might be helpful for others getting started with C++/Envoy development.
FYI The code here is not the greatest and isn't meant to be, but I do plan on coming back here and cleaning up somewhat.
If you don't understand the explanations here, please open an issue/PR. Also, the filter's source code should be well-documented (actually over-documented) to provide further explanations.
git submodule update --init # we need the envoy source to compile/test
make compileThe compile target should call the compile/proto target to generate the protobuf classes in the gen/ directory.
I've included a basic app.py web server that just prints the headers it receives. To run the whole thing:
terminal pane 1:
./bazel-bin/envoy -c examples/example.v2.yaml --v2-config-onlyterminal pane 2:
python examples/app.py
terminal pane 3:
curl -H "x-rot: asdf" localhost:8000
and in pane 2 you should see something like
user-agent: curl/7.54.0
accept: */*
x-forwarded-proto: http
x-request-id: 5a16b07d-d811-4fdf-b9bc-790efdd1d97b
x-rot: nopq
content-length: 0
where you can see the letters of x-rot have been rotated.
This filter accepts two configuration options:
rot_value: an integer describing how much to rotate each letter by. For example, 13 would be the classic Caesar Cipherrot_header: a string that names which header to apply the cipher to. If the header is not found, no action is taken and the filter continues.
v2 Example
http_filters:
- name: rot_cipher
config: { rot_value: 13, rot_header: x-rot }
- name: envoy.router
Full example configurations are in the examples/ directory.
Every filter has 2 main components:
- The config, which for HTTP filters should be a subclass of
NamedHttpFilterConfigFactory - The filter implementation, which for HTTP filters would be a subclass of either
StreamDecoderFilter,StreamEncoderFilter, orStreamFilterfor filters that do encoding and decoding.
To support the v2 API, you'll additionally need a .proto file (if your filter uses configuration data. If it doesn't then there's no need for a .proto file).
Simply put, if your config is v1, then Envoy will call createFilterFactory with a Json::Object which you can then pick apart and validate.
If your config is v2, then it will call createFilterFactoryFromProto with a Protobuf::Message, which you can just cast to your protobuf-generated class.
The details of how/why this procedure works is worthy of a separate write-up, but I believe it has a lot to do with how the v2 API can use gRPC streaming.
(More on Envoy's use of gRPC in the Envoy docs)
If you look at filters in the main Envoy source code they have a lot more pieces due to the complexity, for example they have intermediate config classes that the Json and protobuf data get converted to so that the filters only need 1 constructor. But if you strip it all down, these are the bare minimums.
I started with rot_cipher.proto to define what the configuration would look like.
That file then produces rot_cipher.pb.(cc|h) in gen/.
This is a pretty ad-hoc procedure that I constructed, and quite frankly the bazel BUILD file could be improved to use the native Bazel support for protobufs instead of calling the protobuf compiler in the Makefile.
But is using a .proto necessary?
For the v2 API, yes. You might be able to hack something weird using Protobuf Reflection but that's a scary path.
After wrapping my head around how the protobufs integrate into Envoy filter configs (first-time protobuf user here), the rest was pretty straight forward to achieve based on the envoy-filter-example. Some important classes to look at are HeaderMap for manipulating headers and Object for manipulating Json Objects. Additionally, json_loader.cc, config/utility.h, and protobuf/utility.h were pretty helpful.
As an aside, before starting this filter I went through the Protobuf C++ Tutorial, substituting proto3 syntax for proto2.
(this part is still kind of mumble-y and under revision)
I had an issue with the v2 API config where each time I tried to start Envoy I got the error below:
[2017-12-22 13:55:03.124][4915417][debug][config] external/envoy/source/server/listener_manager_impl.cc:29] filter #0:
[2017-12-22 13:55:03.124][4915417][debug][config] external/envoy/source/server/listener_manager_impl.cc:30] name: envoy.http_connection_manager
[2017-12-22 13:55:03.195][4915417][debug][config] external/envoy/source/server/config/network/http_connection_manager.cc:226] filter #0
[2017-12-22 13:55:03.195][4915417][debug][config] external/envoy/source/server/config/network/http_connection_manager.cc:227] name: rot_cipher
[2017-12-22 13:55:03.195][4915417][critical][main] external/envoy/source/server/server.cc:72] error initializing configuration 'examples/example.v2.yaml': Unable to parse JSON as proto (INVALID_ARGUMENT:: Cannot find field.): {"rot_value":13,"rot_header":"x-rot"}
[2017-12-22 13:55:03.195][4915417][debug][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:523] shutting down thread local cluster manager
I had managed to narrow it down to this line in my code:
const auto& typed_config = dynamic_cast<const example::RotCipher&>(proto_config);All generated protobuf messages subclass Protobuf::Message (which is the type of proto_config) so this dynamic cast shouldn't be a problem. I actually found this snippet in the Envoy source.
I had gone over the configuration multiple times and knew it was correct based on playing with other filters.
At the end my wits, I started throwing print statements into the Envoy source code, thinking maybe there was a bug somewhere in the JSON parsing?
I started with the HttpConnectionManagerConfig right after it logs what filter it's processing, since the debug log kindly outputs what line it is at :)
This is at the end of the constructor which has the following code.
const auto& filters = config.http_filters();
for (int32_t i = 0; i < filters.size(); i++) {
const ProtobufTypes::String& string_name = filters[i].name();
const auto& proto_config = filters[i];
ENVOY_LOG(debug, " filter #{}", i);
ENVOY_LOG(debug, " name: {}", string_name);
const Json::ObjectSharedPtr filter_config =
MessageUtil::getJsonObjectFromMessage(proto_config.config());
// Now see if there is a factory that will accept the config.
auto& factory = Config::Utility::getAndCheckFactory<NamedHttpFilterConfigFactory>(string_name);
HttpFilterFactoryCb callback;
if (filter_config->getBoolean("deprecated_v1", false)) {
callback = factory.createFilterFactory(*filter_config->getObject("value", true),
stats_prefix_, context);
} else {
ProtobufTypes::MessagePtr message =
Config::Utility::translateToFactoryConfig(proto_config, factory);
callback = factory.createFilterFactoryFromProto(*message, stats_prefix_, context);
}
filter_factories_.push_back(callback);
}
I inspected the filter_config JSON object and it had the rot_value and rot_header as expected, so then I looked inside the call to Config::Utility::translateToFactoryConfig(...) which contains the following code.
template <class ProtoMessage, class Factory>
static ProtobufTypes::MessagePtr translateToFactoryConfig(const ProtoMessage& enclosing_message,
Factory& factory) {
ProtobufTypes::MessagePtr config = factory.createEmptyConfigProto();
if (config == nullptr) {
throw EnvoyException(fmt::format(
"{} factory returned nullptr instead of empty config message.", factory.name()));
}
if (enclosing_message.has_config()) {
MessageUtil::jsonConvert(enclosing_message.config(), *config);
}
return config;
}I wasn't getting the nullptr error message so the config wasn't coming back null... WAIT. createEmptyConfigProto? That function is required by the abstract class and I remember pasting in some code for it... I hadn't thought much of that function and just pasted what the envoy-example-filter code had, which is return ProtobufTypes::MessagePtr{new Envoy::ProtobufWkt::Empty()}. If the empty message is not of type example::RotCipher, then yeah there would be issues in casting it to a Protobuf::Message and then to an example::RotCipher. I checked some built-in Envoy filters that had non-empty configurations and sure enough, they return an empty instance of the generated protobuf config.
RateLimit config
ProtobufTypes::MessagePtr createEmptyConfigProto() override {
return ProtobufTypes::MessagePtr{new envoy::api::v2::filter::http::RateLimit()};
}So once I changed the RotCipher config's createEmtpyConfigProto to
ProtobufTypes::MessagePtr createEmptyConfigProto() override {
return ProtobufTypes::MessagePtr{new example::RotCipher()};
}all was well! The error message was a little cryptic (yet correct), but sure enough it was a user-error.
This was mostly an exercise in gaining familiarity with Envoy filter development so here's some thoughts on how the whole thing went.
- There is very little internal documentation of how the mechanics work. This is no secret, but the documentation from Envoy could use some love and help from the community (low-hanging fruit for new Envoy contributors!)
- The organization of the Envoy code made it a little difficult to piece this project together. For example, all the Envoy filter configs are in one directoy (
server/config/http/), and all the Envoy filter implementations are in a different directory (source/common/), and there's multipleconfigdirectories for non-filter config stuff, etc. so it felt like a scavenger hunt to get the complete image of what all 1 filter needs. This organization makes sense for the size of Envoy's project, but again makes it difficult to piece together everything that, for example, the buffer filter does. - Despite the above 2 "complaints", Envoy does provide a very clean API to work with. I found this to be a much easier exercise than implementing an Nginx module, which took a day-long workshop and I'm still not confident in my Nginx module development.
- I noticed there's some scripts in
envoy/tools/such asstack_decode.pythat may have been helpful in debugging but I couldn't figure out how to get them to work. - The bazel build procedure is pretty slick. Props to the Envoy team!
- If you want to write Envoy filters, you can't shy away from reading a lot of C++ :)