Enhance the performance of the frontend JSON codec #6816
Conversation
damnever
force-pushed
the
perf/frontendcodec
branch
3 times, most recently
from
101064f to
6c36bd4
Compare
pkg/querier/tripperware/query.go
Outdated
| @@ -25,6 +25,7 @@ import ( | |||
| "github.com/prometheus/prometheus/util/jsonutil" | |||
| "github.com/weaveworks/common/httpgrpc" | |||
|
|
|||
| _ "github.com/cortexproject/cortex/pkg/chunk" // Register jsoniter type: labels.Labels. | |||
There was a problem hiding this comment.
I am wondering if we have a better package to import? In case we remove some code from that chunk package.
There was a problem hiding this comment.
maybe we can just something like this?
var _ = labels.Labels{}
There was a problem hiding this comment.
maybe we can reimplement the codec for cortexpb.LabelAdapter and use it directly
There was a problem hiding this comment.
I am worried that at sometime we just remove that code path from the package without noticing and we have the regression.
We don't have to do it in this PR but let's at least add some TODO or notes for re-implementing the codec or something similar. Creating an issue also works
There was a problem hiding this comment.
We can export the legacy functions directly and use them without relying on global registration.
pkg/querier/tripperware/query.go
Outdated
| @@ -25,6 +25,7 @@ import ( | |||
| "github.com/prometheus/prometheus/util/jsonutil" | |||
| "github.com/weaveworks/common/httpgrpc" | |||
|
|
|||
| _ "github.com/cortexproject/cortex/pkg/chunk" // Register jsoniter type: labels.Labels. | |||
There was a problem hiding this comment.
maybe we can just something like this?
var _ = labels.Labels{}
|
Just out of curiosity, can you also show the impact in memory on this? |
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
damnever
force-pushed
the
perf/frontendcodec
branch
from
6c36bd4 to
5840e00
Compare
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
damnever
force-pushed
the
perf/frontendcodec
branch
from
08c7800 to
da7343d
Compare
|
@damnever For the graph you shared, when was the change deployed and how much it impacted memory? |
|
@yeya24 the middle of 06/13 12:00 - 06/14 00:00, peak memory decreased significantly, possibly by 70% |
What this PR does:
Stream json encoding using existing functions, reducing CPU usage ~50%.
Which issue(s) this PR fixes:
After upgrading from v1.16 to v1.19, the frontend's CPU usage increased significantly. The primary cause appears to be related to JSON codec, the use of
encoding/jsonand the read-and-parse process. This patch addresses both issues. However, despite applying the patch, CPU usage remains higher than it was prior to the upgrade. The exact reason for this is unclear, maybe it is due to Go 1.24 or something else, but profiling results still show that decoding labels continues to consume a substantial amount of processing time (profile below).Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]