Way to disable logging for certain code paths (scheduled jobs, etc.)

[paveljakov]

Is your feature request related to a problem? Please describe.
Problem we are solving is Java agent logging dependencies of Quartz scheduled jobs. Currently we have periodic jobs that run quite often and perform actions, e.g. perform DB queries, etc. We are generating quite a lot of automatically collected AI dependencies as a result of action performed in these jobs. It would be nice if we could either limit sampling rate or disable automatic log collection for such code paths.

Describe the solution you would like
Few options possible:

1. Add Quarts Job support to AI agent - this would probably generate custom spans for quartz jobs, that we could use to configure sampling overrides on.
2. Or add ability to configure sampling rate programmatically (using SDK or something else), so that it would be possible to limit/disable entire code paths from logging dependencies, etc.
3. Add sampling overrides support for SDK custom telemetry;
4. Provide any other suggestions how to solve periodic jobs dependency telemetry problem.

Describe alternatives you have considered
What we have tried:

• Sampling overrides - kind of works, but you have to explicitly override each and every DB query, etc. Not very manageable solution;
• Create custom parent Span using SDK and use sampling override for parent - sampling override does not work on custom SDK telemetry;
• Create custom parent Span using @WithSpan and use sampling override for parent - Span created with @WithSpan is InProc and does not contain any attributes we could apply sampling overrides on (adding custom attributes with span processor also does not help, because sampling overrides do not work for "added" attributes by span processors).

[trask]

@paveljakov we pulled in Quartz instrumentation as preview instrumentation in the latest release:
https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#preview-instrumentations

[paveljakov]

Hello @trask ,

I have tried new AI agent 3.2.0 G.A. and I see that Quartz telemetry is being collected automatically and dependencies correctly correlated to parent Quartz telemetry element. But I can not seem to find any attributes to apply Sampling Override on.

Current Quartz telemetry presents these properties inside Application Insights transaction details view:

• Event time
• Type
• Call status
• Duration
• Name
• Operation Id
• Parent Id
• Id
• Performance
• Telemetry type
• Application version
• Device type
• Client IP address
• City
• State or province
• Country or region
• Role name
• Role instance
• SDK version
• Sample rate
• Base name

Which of these can be used to apply Sampling Overrides on? And what would be telemetry attribute names for these? I can not find telemetry attribute names for these standard AI properties, some are mentioned here (https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides#common-span-attributes), but they do not apply to more generic (InProc, etc.) telemetry types.

[trask]

@paveljakov great point, there's currently no attributes captured by the quartz instrumentation. I'm proposing to add some in open-telemetry/opentelemetry-java-instrumentation#4332, this would allow you to sample based on the job execution class name. Let me know if you see potential issues with this. Otherwise I can get you a SNAPSHOT build to try out if/when this is accepted upstream.

[paveljakov]

@trask that should work, thank you!

[trask]

Resolved by #1907

@paveljakov here's a snapshot build that you can try out if you have a chance: https://github.com/microsoft/ApplicationInsights-Java/files/7333605/applicationinsights-agent-3.2.1-BETA-SNAPSHOT.jar.zip

[paveljakov]

@trask, We have updated AI agent to newest GA and additional attributes allows us to sample Quartz job execution logging. Thank You!