Observability/Tracing hooks?

[tylerbenson]

I love this new initiative. It shows a lot of promise. One area I see that can be improved is in the observability/traceability front. This includes both API and Agent based.

JDBC's API makes some decisions that complicate these goals. For example, the PreparedStatement interface provides no reference to the original SQL statement. We must maintain a weak map reference to the prepared statement to get that SQL statement.

I'm sure there are other aspects that can be improved here. This is just intended to start that discussion. @adriancole would be a great point of contact within Pivotal to discuss with as well.

[nebhale]

@tylerbenson Can we get together in person for a chat (with @smaldini) to discuss what your overall plan is for observability on Reactive applications is? We'd definitely like to get an idea of how you're going to go about doing it and exactly what you'd like to see as far as features in R2DBC. Drop me an email (bhale@.io) and we'll set something up.

[codefromthecrypt]

Jumping in for some remote pointers as definitely have some experience here okhttp recently added an eventlistener interface which satisfies the needs of metrics and tracing (plus some other things) in one go. Quite a lot of discussion went into this.. I'm not saying it is exactly the same in JDBC but certainly from a hook POV this one is nice. It is nice because it didn't go down the usually bad path or introducing its own "tracer" api (which could need maintenance) or a hard dependency (like opentracing or brave) in such a way that everyone is locked to a 3rd party dep. Low-level interface libraries should be self-contained in other words. https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/EventListener.java Tracing in particular is easier to work with when there is a way to associate some request state from the beginning to the end of the request. Whether that's an attribute bag or the ability to use a weak map, or a well-known lifecycle. For example, again in the above http thing, you can change depth of tracing (eg do a flat thing or have separate stuff for dns), without the listener api having to change. Here's an example https://github.com/yschimke/okurl/blob/master/src/main/kotlin/com/baulsupp/okurl/tracing/ZipkinTracingListener.kt In other words, event listener if done well is usually ideal pattern as it is more flexible than interceptors which usually have only one depth available. On data, things with metrics and tracing is there is a need to scrub the parameters, so knowing a template/prepared statement is nice as it avoids the risk in exposing parameters which would blow up cardinality in indexes or leak things. For example, in http we use a tag "http.route" to allow some knowledge to exist, but avoid some of the data pitfalls. So, in whatever form hooks exist, we should try to make things like templates easy to get a hold of. cc some folks who might either have eventy/tracy/metricsy things to add or are implicitly noted by above @swankjesse @yschimke @trustin @jkschneider @marcingrzejszczak Very excited about observable databasing

…

On Mon, Oct 1, 2018 at 6:30 AM Ben Hale ***@***.***> wrote: @tylerbenson <https://github.com/tylerbenson> Can we get together in person for a chat (with @smaldini <https://github.com/smaldini>) to discuss what your overall plan is for observability on Reactive applications is? We'd definitely like to get an idea of how you're going to go about doing it and exactly what you'd like to see as far as features in R2DBC. Drop me an email ***@***.***) and we'll set something up. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD618IF_KIPcQi0VsI3jm_pOZkPAUiQks5ugUXogaJpZM4W-oLc> .

[marcingrzejszczak]

Can we get together in person for a chat (with @smaldini) to discuss what your overall plan is for observability on Reactive applications is

From the perspective of Spring Cloud Sleuth I can add that we are reusing what @smaldini has added in Reactor in terms of passing Context. If you're using Sleuth and Spring WebFlux things will work out of the box.

[jkschneider]

If you solve the context problem for tracing, you've solved it for metrics. I'd love to see a preliminary interface when you have one in place so I can take it a spin for metrics instrumentation.

[marcingrzejszczak]

If you solve the context problem for tracing, you've solved it for metrics. I'd love to see a preliminary interface when you have one in place so I can take it a spin for metrics instrumentation.

Yeah, so I'm telling you that, cause Sleuth does work with Reactor :)

[jkschneider]

Yeah, so I'm telling you that, cause Sleuth does work with Reactor :)

If you mean to suggest that we haven't completed metrics support for Reactor, they actually embedded it directly into reactor-core ;). reactor/reactor-core#1123

[ttddyy]

Hello,

I write a framework called datasource-proxy which provides listeners for JDBC API using proxy.

The listener API provides:

• Query logging
• Slow query detection
• Method tracing
• Metrics
• Assertion/Verification
  • Connection leak detection
  • Transaction check
• Custom logic injection
• etc.

I recently started similar for r2dbc - datasource-proxy-r2dbc.

datasource-proxy-r2dbc provides a listener that receives:

• before/after any method invocation on SPI
• before/after query execution
• processing each query result

Currently, I'm creating proxy for R2DBC SPI classes and intercepting all method calls. Based on the invoked method, it triggers the registered listeners with contextual information as method parameter.

Since this thread is about observability/traceability, I'd like to also suggest some API from my library perspective.

Mainly, I use two events via proxy:

• Execution of every method on ConnectionFactory, Connection, Batch, and Statement(for now)
• Execution of query method (Batch#execute(), Statement#execute())

Method execution

I use proxy for intercept SPI interaction; however, if following API exists on SPI, I believe ultimately it solves a lot of observability/traceability related tasks.

// called back on any method invocation on SPI
void onMethodExecution(MethodInvocationContext context);

// if we allow value replacement, then maybe something like:
// Object onMethodExecution(MethodInvocationContext context);

This callback API is essentially what proxy does.
In datasource-proxy-r2dbc, I define MethodExecutionInfo class for contextual information and provide it to a listener callback parameter.

This is a low level callback with all related information packed in a parameter. If we want to have smart listener which understands connection open/close, transaction start/commit/rollback, batching, parameter bindings, savepoints, then such listener may be built on top of this method. (Or, simply each SPI method may invoke listener)

I think this is great for library; however, this may impose more on driver implementation because it needs to call/notify the every SPI invocation to the callback method.

Query execution

Executing query is a multi step operation - obtain connection, create statement, bind parameters, execute, retrieve results.

datasource-proxy provides listener callback for query execution with contextual information such as batched queries and parameters bound to the execution by keeping track of such multi-step-operations.

callback API would be something like this:

// called back on Batch#exeucte() or Statement#execute()
void onQueryExecution(QueryExecutionContext context);

In datasource-proxy-r2dbc, I define QueryExecutionInfo to provide contextual information.

By looking r2dbc-[postgres|h2] impl, this might be easier to implement.
Just compose query execution contextual information which r2dbc-driver must already be aware of, then call the callback method on listener.
Also, in order to expose contextual information, Bindings related classes need to be added to SPI as well.

So, I'm just putting down my idea for listener/callback on SPI here.
For datasource-proxy and maybe some other library works on this layer, I think method/query callbacks are essential part of the framework.

Thanks,

[codefromthecrypt]

well thought through and written. probably in brave we could use your approach/library.. worth a test integration I think.

…

On Sat, 3 Nov 2018, 14:54 Tadaya Tsuyukubo, ***@***.***> wrote: Hello, I write a framework called datasource-proxy <https://github.com/ttddyy/datasource-proxy> which provides listeners for JDBC API using proxy. The listener API provides: - Query logging - Slow query detection - Method tracing - Metrics - Assertion/Verification - Connection leak detection - Transaction check - Custom logic injection - etc. I recently started similar for r2dbc - datasource-proxy-r2dbc <https://github.com/ttddyy/datasource-proxy-r2dbc>. datasource-proxy-r2dbc provides a listener that receives: - before/after any method invocation on SPI - before/after query execution - processing each query result Currently, I'm creating proxy for R2DBC SPI classes and intercepting all method calls. Based on the invoked method, it triggers the registered listeners with contextual information as method parameter. Since this thread is about observability/traceability, I'd like to also suggest some API from my library perspective. Mainly, I use two events via proxy: - Execution of every method on ConnectionFactory, Connection, Batch, and Statement(for now) - Execution of query method (Batch#execute(), Statement#execute()) Method execution I use proxy for intercept SPI interaction; however, if following API exists on SPI, I believe ultimately it solves a lot of observability/traceability related tasks. // called back on any method invocation on SPIvoid onMethodExecution(MethodInvocationContext context); // if we allow value replacement, then maybe something like:// Object onMethodExecution(MethodInvocationContext context); This callback API is essentially what proxy does. In datasource-proxy-r2dbc, I define MethodExecutionInfo <https://github.com/ttddyy/datasource-proxy-r2dbc/blob/master/src/main/java/net/ttddyy/dsproxy/r2dbc/core/MethodExecutionInfo.java> class for contextual information and provide it to a listener callback parameter. This is a low level callback with all related information packed in a parameter. If we want to have smart listener which understands connection open/close, transaction start/commit/rollback, batching, parameter bindings, savepoints, then such listener may be built on top of this method. (Or, simply each SPI method may invoke listener) I think this is great for library; however, this may impose more on driver implementation because it needs to call/notify the every SPI invocation to the callback method. Query execution Executing query is a multi step operation - obtain connection, create statement, bind parameters, execute, retrieve results. datasource-proxy provides listener callback for query execution with contextual information such as batched queries and parameters bound to the execution by keeping track of such multi-step-operations. callback API would be something like this: // called back on Batch#exeucte() or Statement#execute()void onQueryExecution(QueryExecutionContext context); In datasource-proxy-r2dbc, I define QueryExecutionInfo <https://github.com/ttddyy/datasource-proxy-r2dbc/blob/master/src/main/java/net/ttddyy/dsproxy/r2dbc/core/QueryExecutionInfo.java> to provide contextual information. By looking r2dbc-[postgres|h2] impl, this might be easier to implement. Just compose query execution contextual information which r2dbc-driver must already be aware of, then call the callback method on listener. Also, in order to expose contextual information, Bindings related classes need to be added to SPI as well. So, I'm just putting down my idea for listener/callback on SPI here. For datasource-proxy and maybe some other library works on this layer, I think method/query callbacks are essential part of the framework. Thanks, — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD613T58qfEwQ2MR4K-d-oILhJpLvcoks5urT28gaJpZM4W-oLc> .

[ttddyy]

Regarding implementing callback/listener/extension-points, I'd like to suggest creating another layer on top of r2dbc-SPI.

I believe key focus is different between callback/listener users(e.g.: tracing/metrics/logging) and r2dbc-driver (which implements SPI).

For r2dbc-driver, the main focus is to make it work with specific database.
For tracing/metrics/connection-pool/etc, their focus is to do something based on the user interaction on SPI.

In order to implement callback/listener/extension-points for generic SPI, enforcing each individual r2dbc-driver to implement such callback may not be optimal.
For example, implementing onMethod() callback (called when each SPI is invoked) may not be what r2dbc-driver focuses on, rather it does not want to implement such callback in driver.

So, instead of defining generic callback/listener/extension-points in r2dbc-spi and letting each r2dbc-driver implement it, I suggest creating another layer on top of r2dbc-spi that provides callback/listener/extension-points on SPI interaction. (maybe called r2dbc-proxy, r2dbc-callback, r2dbc-extension, etc)
Then, integration with tracing, metrics, logging, (connection-pooling), etc. can happen on this layer
This layer is essentially what I do on datasource-proxy-r2dbc. (I can donate my code if go this route)

By separating callback layer and SPI:

• r2dbc driver does NOT need to know or implement about generic listeners/callbacks
  • They can still write listeners/callbacks for implementation specific items. For example, some callbacks for postgres specific implementation, so that metrics can be updated, etc.
• clients(r2dbc-client, spring-data-r2dbc) do NOT need to know how listeners are implemented since they work on r2dbc-SPI (or, they may use callback/listeners)
• Something works on generic SPI(tracing, metrics, logging, connection-pooling, etc) can be implemented on this layer.

Sample code:

PostgresConnectionFactory originalConnectionFactory = ...

ConnectionFactory cf = 
  ConnectionFactoryBuilder
    .from(originalConnectionFactory)
    .listener(new TracingListener())  // integrate with tracing
    .listener(new MetricsListener())  // integrate with metrics
    .listener(new LoggingListener())  // integrate with logging
    .listener(new MyConnectionPoolingListener())  // maybe connection-pooling might work similarly
    .build();

R2dbc client = new R2dbc(cf);

cc @nebhale

[ttddyy]

Hi,
I have released v0.1 of datasource-proxy-r2dbc (uses r2dbc-1.0.0.M6).
As an example, I have created a sample integration for tracing using Sleuth.

For tech detail:
tracing is implemented in TracingExecutionListener.
This listener implements LifeCycleListener which defines before/after callbacks for all methods of proxied SPI classes. This is achieved by LifeCycleExecutionListener using proxy which intercept all method calls.