[SDTEST-409] Send telemetry events in batches

[anmarchenko]

What does this PR do?
Optimises telemetry events sending by sending them in batches using message-batch event type.

Motivation:
I expect the number of telemetry events to grow as soon as products start using metrics. We can save time by batching the list of events into a single event.

How to test the change?
System tests pass for telemetry events.

Logs from rails app with DD_TRACE_DEBUG=1:

D, [2024-07-01T16:08:58.970222 #20478] DEBUG -- datadog: [datadog] (/Users/andrey.marchenko/p/dd-trace-rb/lib/datadog/core/telemetry/worker.rb:88:in `flush_events') Sending 2 telemetry events
D, [2024-07-01T16:08:59.137402 #20478] DEBUG -- datadog: [datadog] (/Users/andrey.marchenko/p/dd-trace-rb/lib/datadog/core/telemetry/emitter.rb:29:in `request') Telemetry sent for event `message-batch` (code: 202)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 98.64865% with 1 line in your changes missing coverage. Please review.

Project coverage is 98.11%. Comparing base (4cd328d) to head (8d02f3e).
Report is 30 commits behind head on master.

Files	Patch %	Lines
lib/datadog/core/telemetry/event.rb	97.67%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3749   +/-   ##
=======================================
  Coverage   98.11%   98.11%           
=======================================
  Files        1235     1235           
  Lines       73749    73788   +39     
  Branches     3560     3562    +2     
=======================================
+ Hits        72357    72398   +41     
+ Misses       1392     1390    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[p-datadog]

Is it OK that seq id is the same for all of the events?

[anmarchenko]

Great question, I need to look it up in the spec.

[anmarchenko]

The spec says nothing about it and also it does not mention that seq_id should be part of telemetry events payloads at all (it is supposed to be part of common headers)! I will investigate why our payload generation code even depends on seq_id...

[anmarchenko]

I found where this seq_id is defined in specs: https://github.com/DataDog/instrumentation-telemetry-api-docs/blob/main/GeneratedDocumentation/ApiDocs/v2/SchemaDocumentation/Schemas/conf_key_value.md

This field should incremented each time a new configuration key-value pair is applied in order to track the > active set of configurations. It can be a global value across all configurations keys or local to each key..

This seq_id is in fact different from the seq_id that is used for events:
https://github.com/DataDog/instrumentation-telemetry-api-docs/blob/main/GeneratedDocumentation/ApiDocs/v2/development.md#required-top-level-json-keys

Counter that should be auto incremented every time an API call is being made (our pipeline may not always work in a sequential order.This field will help us identify situations where some messages were being lost. seq_id should start at 1 so that the go backend can detect between non set fields.

I will see how to decouple them from each other in our implementation

[anmarchenko]

Solved this by introducing another sequence in Core::Telemetry::Event class for the configuration changes