Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial OpenCensus tracing support. #5387

Merged
merged 80 commits into from
Jun 5, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
e517ebf
Add initial OpenCensus tracing support.
g-easy Dec 10, 2018
de6b334
tools/check_repositories.sh: make regexes look for word start.
g-easy Dec 21, 2018
0c17e0a
Fix clang_tidy error.
g-easy Dec 21, 2018
461d9b0
Merge remote-tracking branch 'origin/master' into oc
g-easy Jan 18, 2019
0bcaf23
Merge remote-tracking branch 'origin/master' into oc
g-easy Jan 22, 2019
ec0f25d
Fix mismerge.
g-easy Jan 22, 2019
2f0d285
Remove debugging printfs.
g-easy Jan 22, 2019
d96f105
Add explicit break.
g-easy Jan 22, 2019
0e28684
Format fix.
g-easy Jan 22, 2019
e54cadb
Add opencensus:config_test.
g-easy Jan 22, 2019
1081de3
Don't apply empty TraceConfig.
g-easy Jan 22, 2019
471dc28
Add opencensus:tracer_test.
g-easy Jan 22, 2019
e5d3b9a
clang_tidy fixes.
g-easy Jan 29, 2019
c5d46f4
Merge remote-tracking branch 'origin/master' into oc
g-easy Feb 1, 2019
7e444e3
Mark config proto experimental.
g-easy Feb 1, 2019
0902d7d
Use OpenCensus propagation helpers for traceparent, grpc-trace-bin.
g-easy Feb 1, 2019
710a6a4
Don't use glob() in curl.BUILD.
g-easy Feb 1, 2019
886bccb
Merge remote-tracking branch 'origin/master' into oc
g-easy Feb 7, 2019
06e0d1e
Merge remote-tracking branch 'origin/master' into oc
g-easy Feb 22, 2019
bd901c5
Bump date.
g-easy Feb 22, 2019
024611d
Caps.
g-easy Feb 22, 2019
8f0a4ff
Use camelCase, add comments about interplay with exporter thread.
g-easy Feb 22, 2019
916b860
Merge remote-tracking branch 'origin/master' into oc
g-easy Feb 25, 2019
a984a60
Use ENVOY_LOG_MISC for debug string.
g-easy Feb 25, 2019
38a19c6
Variable names.
g-easy Feb 25, 2019
78d3320
Make samplers non-static.
g-easy Feb 25, 2019
16a3e3d
Remove underscore.
g-easy Feb 27, 2019
16d9939
Delete commented out leftover.
g-easy Feb 27, 2019
cdc9b3c
Comment.
g-easy Feb 27, 2019
7bbdce8
Merge remote-tracking branch 'origin/master' into oc
g-easy Feb 27, 2019
b19efcb
Upgrade opencensus-cpp dep.
g-easy Feb 27, 2019
9a1c637
Better example of trace_config.
g-easy Feb 27, 2019
cd0486b
Reset TraceParams back to default.
g-easy Feb 27, 2019
85d4e77
format
g-easy Feb 27, 2019
93846d4
Merge remote-tracking branch 'origin/master' into oc
g-easy Mar 12, 2019
a9e19df
Add log()
g-easy Mar 12, 2019
40f4308
Update opencensus-proto to catch trace_config.proto change.
g-easy Mar 12, 2019
ec14fd5
Update ConstantSampler handling.
g-easy Mar 12, 2019
d65ebb0
Use envoy_cmake_external for curl.
g-easy Mar 12, 2019
e17a406
Update config and tracer test.
g-easy Mar 12, 2019
f7a90b3
Fix compile_time_options: set CMAKE_BUILD_TYPE for curl.
g-easy Mar 12, 2019
8eea063
Optionally accept and propagate an incoming trace context.
g-easy Mar 12, 2019
26853eb
Fix config test.
g-easy Mar 12, 2019
0afc3eb
Fix envoy.api.
g-easy Mar 12, 2019
e5c743e
Merge remote-tracking branch 'origin/master' into oc
g-easy Mar 19, 2019
9efab49
Upgrade and correctly name opencensus_proto.
g-easy Mar 19, 2019
fd739ca
clang-tidy fixup.
g-easy Mar 19, 2019
19dbc58
Fix build:go_build_test
g-easy Apr 11, 2019
1c56a62
Merge remote-tracking branch 'origin/master' into oc
g-easy Apr 11, 2019
c06ae5f
Merge remote-tracking branch 'origin/master' into oc
g-easy Apr 15, 2019
a7f53f2
Test constant_sampler and more of the config.
g-easy Apr 15, 2019
5b78b82
format
g-easy Apr 15, 2019
6a3e7db
Test spawnChild().
g-easy Apr 15, 2019
0b0f6df
Test injectContext().
g-easy Apr 15, 2019
8ab8440
format
g-easy Apr 15, 2019
db9f80f
Merge remote-tracking branch 'origin/master' into oc
g-easy Apr 18, 2019
199a060
Merge remote-tracking branch 'origin/master' into oc
g-easy May 20, 2019
f2ca81e
Upgrade OpenCensus C++ to work around bazel build error.
g-easy May 20, 2019
6ccab02
Need to also upgrade absl to get new flag name.
g-easy May 20, 2019
1fb1b9c
Catch up to new API.
g-easy May 20, 2019
142a386
Fix libcurl linkage.
g-easy May 23, 2019
ee4d4d0
Merge remote-tracking branch 'origin/master' into oc
g-easy Jun 3, 2019
41f7dff
Change of direction: don't upgrade absl.
g-easy Jun 3, 2019
bbaca7f
Patch opencensus to work with current absl.
g-easy Jun 3, 2019
7d07e53
Adapt to commit 8d1ad35aa7
g-easy Jun 3, 2019
a4e7038
Avoid ODR by not using lightstep_vendored_googleapis.
g-easy May 29, 2019
4719582
Switch custom googleapis build to the native cc_proto_library rule.
g-easy Jun 4, 2019
154b58c
Add @googleapis//:tracing_proto for stackdriver trace.
g-easy Jun 4, 2019
88e6a19
curl without libssh2, trying to fix macos build.
g-easy Jun 4, 2019
f587ee8
Review comments.
g-easy Jun 4, 2019
b87a8e1
grpc-trace-bin needs to be base64 encoded/decoded without padding.
g-easy Jun 4, 2019
9ca882f
Finish todo: tests for grpc-trace-bin.
g-easy Jun 4, 2019
c8fc2c9
Spellcheck doesn't like the comment.
g-easy Jun 4, 2019
03df4ad
Add decode_without_padding().
g-easy Jun 4, 2019
2d9e96b
Use decode_without_padding().
g-easy Jun 4, 2019
9d89449
decodeWithoutPadding rename + take string_view
g-easy Jun 5, 2019
4e3245c
Make ALWAYS_ON fallthrough.
g-easy Jun 5, 2019
4122637
Current span context is always valid.
g-easy Jun 5, 2019
3b9ced4
Factor out SamplerTestHelper and increase coverage.
g-easy Jun 5, 2019
7385873
speling
g-easy Jun 5, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 37 additions & 25 deletions api/bazel/repositories.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,8 @@ def api_dependencies():
build_file_content = PROMETHEUSMETRICS_BUILD_CONTENT,
)
envoy_http_archive(
name = "io_opencensus_trace",
name = "opencensus_proto",
locations = REPOSITORY_LOCATIONS,
build_file_content = OPENCENSUSTRACE_BUILD_CONTENT,
)
envoy_http_archive(
name = "kafka_source",
Expand All @@ -38,8 +37,9 @@ def api_dependencies():
)

GOOGLEAPIS_BUILD_CONTENT = """
load("@com_google_protobuf//:protobuf.bzl", "cc_proto_library", "py_proto_library")
load("@com_google_protobuf//:protobuf.bzl", "py_proto_library")
load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")
load("@com_github_grpc_grpc//bazel:cc_grpc_library.bzl", "cc_grpc_library")

filegroup(
name = "api_httpbody_protos_src",
Expand All @@ -58,12 +58,7 @@ proto_library(

cc_proto_library(
name = "api_httpbody_protos",
srcs = [
"google/api/httpbody.proto",
],
default_runtime = "@com_google_protobuf//:protobuf",
protoc = "@com_google_protobuf//:protoc",
deps = ["@com_google_protobuf//:cc_wkt_protos"],
deps = [":api_httpbody_protos_proto"],
visibility = ["//visibility:public"],
)

Expand Down Expand Up @@ -114,14 +109,7 @@ proto_library(

cc_proto_library(
name = "http_api_protos",
srcs = [
"google/api/annotations.proto",
"google/api/http.proto",
],
default_runtime = "@com_google_protobuf//:protobuf",
protoc = "@com_google_protobuf//:protoc",
linkstatic = 1,
deps = ["@com_google_protobuf//:cc_wkt_protos"],
deps = [":http_api_protos_proto"],
visibility = ["//visibility:public"],
)

Expand Down Expand Up @@ -164,14 +152,9 @@ proto_library(
)

cc_proto_library(
name = "rpc_status_protos",
srcs = ["google/rpc/status.proto"],
default_runtime = "@com_google_protobuf//:protobuf",
protoc = "@com_google_protobuf//:protoc",
deps = [
"@com_google_protobuf//:cc_wkt_protos"
],
visibility = ["//visibility:public"],
name = "rpc_status_protos",
deps = [":rpc_status_protos_lib"],
visibility = ["//visibility:public"],
)

go_proto_library(
Expand All @@ -195,6 +178,35 @@ py_proto_library(
visibility = ["//visibility:public"],
deps = ["@com_google_protobuf//:protobuf_python"],
)

proto_library(
name = "tracing_proto_proto",
srcs = [
"google/devtools/cloudtrace/v2/trace.proto",
"google/devtools/cloudtrace/v2/tracing.proto",
],
deps = [
":http_api_protos_proto",
":rpc_status_protos_lib",
"@com_google_protobuf//:timestamp_proto",
"@com_google_protobuf//:wrappers_proto",
"@com_google_protobuf//:empty_proto",
],
)

cc_proto_library(
name = "tracing_proto_cc",
deps = [":tracing_proto_proto"],
)

cc_grpc_library(
name = "tracing_proto",
srcs = [":tracing_proto_proto"],
deps = [":tracing_proto_cc"],
grpc_only = True,
visibility = ["@io_opencensus_cpp//opencensus:__subpackages__"],
)

"""

GOGOPROTO_BUILD_CONTENT = """
Expand Down
12 changes: 6 additions & 6 deletions api/bazel/repository_locations.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ BAZEL_SKYLIB_SHA256 = "2ef429f5d7ce7111263289644d233707dba35e39696377ebab8b0bc70
GOGOPROTO_RELEASE = "1.2.1"
GOGOPROTO_SHA256 = "99e423905ba8921e86817607a5294ffeedb66fdd4a85efce5eb2848f715fdb3a"

OPENCENSUS_RELEASE = "0.1.0"
OPENCENSUS_SHA256 = "4fd21cc6de63d7cb979fd749d8101ff425905aa0826fed26019d1c311fcf19a7"
OPENCENSUS_PROTO_GIT_SHA = "d5d80953a8c2ff4633087af6933cd152678434bb" # Mar 18, 2019
OPENCENSUS_PROTO_SHA256 = "a4e87a1da21d1b3a16674332c3ee6e2689d52f3532e2ff8cb4a626c8bcdabcfc"

PGV_GIT_SHA = "26db5cb7c01a67c6a2e21a832106406185530b2f"
PGV_SHA256 = "6510cbcf69d99059c652ae2376f6240bc761d0b019cd962225f4f609be361e26"
Expand Down Expand Up @@ -44,10 +44,10 @@ REPOSITORY_LOCATIONS = dict(
strip_prefix = "client_model-" + PROMETHEUS_GIT_SHA,
urls = ["https://github.com/prometheus/client_model/archive/" + PROMETHEUS_GIT_SHA + ".tar.gz"],
),
io_opencensus_trace = dict(
sha256 = OPENCENSUS_SHA256,
strip_prefix = "opencensus-proto-" + OPENCENSUS_RELEASE + "/src/opencensus/proto/trace/v1",
urls = ["https://github.com/census-instrumentation/opencensus-proto/archive/v" + OPENCENSUS_RELEASE + ".tar.gz"],
opencensus_proto = dict(
sha256 = OPENCENSUS_PROTO_SHA256,
strip_prefix = "opencensus-proto-" + OPENCENSUS_PROTO_GIT_SHA + "/src",
urls = ["https://github.com/census-instrumentation/opencensus-proto/archive/" + OPENCENSUS_PROTO_GIT_SHA + ".tar.gz"],
),
kafka_source = dict(
sha256 = KAFKA_SOURCE_SHA,
Expand Down
2 changes: 2 additions & 0 deletions api/envoy/config/trace/v2/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ api_proto_library_internal(
],
deps = [
"//envoy/api/v2/core:grpc_service",
"@opencensus_proto//opencensus/proto/trace/v1:trace_config_proto",
],
)

Expand All @@ -18,5 +19,6 @@ api_go_proto_library(
proto = ":trace",
deps = [
"//envoy/api/v2/core:grpc_service_go_proto",
"@opencensus_proto//opencensus/proto/trace/v1:trace_and_config_proto_go",
],
)
49 changes: 49 additions & 0 deletions api/envoy/config/trace/v2/trace.proto
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ option java_package = "io.envoyproxy.envoy.config.trace.v2";
option go_package = "v2";

import "envoy/api/v2/core/grpc_service.proto";
import "opencensus/proto/trace/v1/trace_config.proto";

import "google/protobuf/any.proto";
import "google/protobuf/struct.proto";
Expand All @@ -33,6 +34,7 @@ message Tracing {
// - *envoy.zipkin*
// - *envoy.dynamic.ot*
// - *envoy.tracers.datadog*
// - *envoy.tracers.opencensus*
string name = 1 [(validate.rules).string.min_bytes = 1];

// Trace driver specific configuration which depends on the driver being instantiated.
Expand All @@ -42,6 +44,7 @@ message Tracing {
// - :ref:`ZipkinConfig <envoy_api_msg_config.trace.v2.ZipkinConfig>`
// - :ref:`DynamicOtConfig <envoy_api_msg_config.trace.v2.DynamicOtConfig>`
// - :ref:`DatadogConfig <envoy_api_msg_config.trace.v2.DatadogConfig>`
// - :ref:`OpenCensusConfig <envoy_api_msg_config.trace.v2.OpenCensusConfig>`
oneof config_type {
google.protobuf.Struct config = 2;

Expand Down Expand Up @@ -103,6 +106,52 @@ message DatadogConfig {
string service_name = 2 [(validate.rules).string.min_bytes = 1];
}

// Configuration for the OpenCensus tracer.
// [#proto-status: experimental]
message OpenCensusConfig {
// Configures tracing, e.g. the sampler, max number of annotations, etc.
opencensus.proto.trace.v1.TraceConfig trace_config = 1;

// Enables the stdout exporter if set to true. This is intended for debugging
// purposes.
bool stdout_exporter_enabled = 2;

// Enables the Stackdriver exporter if set to true. The project_id must also
// be set.
bool stackdriver_exporter_enabled = 3;

// The Cloud project_id to use for Stackdriver tracing.
string stackdriver_project_id = 4;

// Enables the Zipkin exporter if set to true. The url and service name must
// also be set.
bool zipkin_exporter_enabled = 5;

// The URL to Zipkin, e.g. "http://127.0.0.1:9411/api/v2/spans"
string zipkin_url = 6;
htuch marked this conversation as resolved.
Show resolved Hide resolved

// The Zipkin service name.
string zipkin_service_name = 7;
g-easy marked this conversation as resolved.
Show resolved Hide resolved

enum TraceContext {
// W3C Trace-Context format "traceparent:" header.
trace_context = 0;

// Binary "grpc-trace-bin:" header.
grpc_trace_bin = 1;

// "X-Cloud-Trace-Context:" header.
cloud_trace_context = 2;
}

// List of incoming trace context headers we will accept. First one found
// wins.
repeated TraceContext incoming_trace_context = 8;

// List of outgoing trace context headers we will produce.
repeated TraceContext outgoing_trace_context = 9;
}

// Configuration structure.
message TraceServiceConfig {
// The upstream gRPC cluster that hosts the metrics service.
Expand Down
4 changes: 2 additions & 2 deletions api/envoy/service/trace/v2/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ api_proto_library_internal(
require_py = 0,
deps = [
"//envoy/api/v2/core:base",
"@io_opencensus_trace//:trace_model",
"@opencensus_proto//opencensus/proto/trace/v1:trace_proto",
],
)

Expand All @@ -18,6 +18,6 @@ api_go_grpc_library(
proto = ":trace_service",
deps = [
"//envoy/api/v2/core:base_go_proto",
"@io_opencensus_trace//:trace_model_go_proto",
"@opencensus_proto//opencensus/proto/trace/v1:trace_proto_go",
],
)
2 changes: 1 addition & 1 deletion api/envoy/service/trace/v2/trace_service.proto
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ option go_package = "v2";
option java_generic_services = true;

import "envoy/api/v2/core/base.proto";
import "trace.proto";
import "opencensus/proto/trace/v1/trace.proto";

import "google/api/annotations.proto";

Expand Down
1 change: 1 addition & 0 deletions bazel/external/rapidjson.BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ licenses(["notice"]) # Apache 2
cc_library(
name = "rapidjson",
hdrs = glob(["include/rapidjson/**/*.h"]),
defines = ["RAPIDJSON_HAS_STDSTRING=1"],
includes = ["include"],
visibility = ["//visibility:public"],
)
20 changes: 20 additions & 0 deletions bazel/foreign_cc/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,26 @@ envoy_cmake_external(
}),
)

envoy_cmake_external(
name = "curl",
cache_entries = {
"BUILD_CURL_EXE": "off",
"BUILD_SHARED_LIBS": "off",
"CMAKE_BUILD_TYPE": "RelWithDebInfo",
"HTTP_ONLY": "on",
"BUILD_TESTING": "off",
"CMAKE_USE_OPENSSL": "off",
"CURL_CA_PATH": "none",
"CURL_HIDDEN_SYMBOLS": "off",
"CMAKE_USE_LIBSSH2": "off",
},
lib_source = "@com_github_curl//:all",
static_libraries = select({
"//bazel:windows_x86_64": ["curl.lib"],
"//conditions:default": ["libcurl.a"],
}),
)

envoy_cmake_external(
name = "event",
cache_entries = {
Expand Down
11 changes: 11 additions & 0 deletions bazel/foreign_cc/com_lightstep_tracer_cpp.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
--- lightstep-tracer-common/BUILD 2019-05-29 13:04:55.651083767 -0700
+++ lightstep-tracer-common/BUILD 2019-05-29 13:04:59.711080560 -0700
@@ -2,7 +2,7 @@
name = "collector_proto",
srcs = ["collector.proto"],
deps = [
- "@lightstep_vendored_googleapis//:googleapis_proto",
+ "@googleapis//:http_api_protos_proto",
"@com_google_protobuf//:timestamp_proto",
],
visibility = ["//visibility:public"],
52 changes: 52 additions & 0 deletions bazel/foreign_cc/io_opencensus_cpp.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
--- copts.bzl 2019-06-03 16:42:49.752903373 -0700
htuch marked this conversation as resolved.
Show resolved Hide resolved
+++ opencensus/copts.bzl 2019-06-03 16:43:58.136847278 -0700
@@ -22,25 +22,25 @@
"""

load(
- "@com_google_absl//absl:copts/GENERATED_copts.bzl",
- "ABSL_GCC_FLAGS",
- "ABSL_GCC_TEST_FLAGS",
- "ABSL_LLVM_FLAGS",
- "ABSL_LLVM_TEST_FLAGS",
- "ABSL_MSVC_FLAGS",
- "ABSL_MSVC_TEST_FLAGS",
+ "@com_google_absl//absl:copts.bzl",
+ "GCC_FLAGS",
+ "GCC_TEST_FLAGS",
+ "LLVM_FLAGS",
+ "LLVM_TEST_FLAGS",
+ "MSVC_FLAGS",
+ "MSVC_TEST_FLAGS",
)

WERROR = ["-Werror=return-type", "-Werror=switch"]

DEFAULT_COPTS = select({
- "//opencensus:llvm_compiler": ABSL_LLVM_FLAGS + WERROR,
- "//opencensus:windows": ABSL_MSVC_FLAGS,
- "//conditions:default": ABSL_GCC_FLAGS + WERROR,
+ "//opencensus:llvm_compiler": LLVM_FLAGS + WERROR,
+ "//opencensus:windows": MSVC_FLAGS,
+ "//conditions:default": GCC_FLAGS + WERROR,
})

TEST_COPTS = DEFAULT_COPTS + select({
- "//opencensus:llvm_compiler": ABSL_LLVM_TEST_FLAGS + WERROR,
- "//opencensus:windows": ABSL_MSVC_TEST_FLAGS,
- "//conditions:default": ABSL_GCC_TEST_FLAGS + WERROR,
+ "//opencensus:llvm_compiler": LLVM_TEST_FLAGS + WERROR,
+ "//opencensus:windows": MSVC_TEST_FLAGS,
+ "//conditions:default": GCC_TEST_FLAGS + WERROR,
})
--- BUILD 2019-06-03 18:30:58.511651926 -0700
+++ opencensus/exporters/trace/stackdriver/BUILD 2019-06-03 18:32:38.107571186 -0700
@@ -28,7 +28,7 @@
copts = DEFAULT_COPTS,
visibility = ["//visibility:public"],
deps = [
- "//google/devtools/cloudtrace/v2:tracing_proto",
+ "@googleapis//:tracing_proto",
Copy link
Contributor Author

@g-easy g-easy Jun 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[in future we can drop this patch] except for this bit. We have to use a single tracing_proto target to avoid ODR (with lightstep).

The long term fix for this is to upgrade googleapis, but first we need to add Python language support for googleapis, but first we need an updated py_proto_library rule in bazel, per:

# TODO(htuch): Convert this to native py_proto_library once
# https://github.com/bazelbuild/bazel/issues/3935 and/or
# https://github.com/bazelbuild/bazel/issues/2626 are resolved.
def api_py_proto_library(name, srcs = [], deps = [], has_services = 0):
py_proto_library(

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also: grpc/grpc#19183

"//opencensus/common:version",
"//opencensus/common/internal/grpc:status",
"//opencensus/common/internal/grpc:with_user_agent",
Loading