"cannot unmarshal log fields" issues after migrating to v0.3.0

[TheYkk]

Describe the bug

VictoriaTraces/app/vtselect/traces/jaeger/jaeger.go:247 cannot unmarshal log fields [DATA HERE] to span: invalid event field: event:0:event_dropped_attributes_count

To Reproduce

trace data with event

Version

v0.3.0

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

[TheYkk]

Also after v0.3 upgrade the memory usage jumped from 6% to 15%

Thread count exploded as well

[jiekun]

Hello! Thanks for reporting the issue. Let me try to explain before diving deeper to the possible solution:

• It's likely that the data are ingested by old version (< v0.1.0).
  • Before the initial release (v0.1.0), the event data will be persisted like: event:{index}:{xxxx}.
  • Since v0.1.0, the event data has been changed to event:{xxxx}:{index}.

We cannot classify this as a bug, as the change was made prior to our initial release, and we cannot provide data compatibility promise for that.

Regarding the memory usage change, it's likely related to #22, which added in-memory cache for index generation. Could you share the following info as well, so that we could know if the difference is expected:

1. The trace span ingestion rate.
2. The total available memory.
3. The mem profile before and after the version upgrade, which should be available on http://127.0.0.1:10428/debug/pprof/heap.

[TheYkk]

The new version pprof: https://pprof.me/c3035fcecbf7826654012a9b187ee4a3/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes&color_by=filename

[jiekun]

That looks like a big increase in memory usage. Based on the profile of v0.3.0, the cache of trace_id doesn't seem to affect the usage much. By default, the traceID cache requires only 32 MiB.

I am currently unable to diagnose this issue. If possible, I will continue to work on the performance optimization issues after the release of v0.4.0. Query performance and resource usage remain our top priorities, and we are committed to continuing the optimization efforts.

[jiekun]

Close as answered.