Sort ingest pipeline stats by time spent executing #88035
Conversation
This commit changes the /_nodes/stats API to return the pipelines in descending order by the count
of how often they are used. If a pipeline ties another pipeline, they are sorted by ingest time.
This ensures that the most used and most expensive pipelines show up first in the JSON response (to
help tracking down which pipelines may be most prevalent).
For example:
```json
{
"nodes" : {
"yiv09_ETRKSv4q8kz0d59A" : {
...
"ingest" : {
"total" : {...},
"pipelines" : {
"set-name" : {
"count" : 4, // <-- sorted first by this
"time_in_millis" : 3, // <-- then by this
...
},
"set-face" : {
"count" : 2, // <-- less than 4
"time_in_millis" : 8,
...
}
```
dakrone
added
>enhancement
:Data Management/Stats
elasticmachine
added
the
Team:Data Management
|
Pinging @elastic/es-data-management (Team:Data Management) |
|
Hi @dakrone, I've created a changelog YAML for you. |
|
@elasticmachine run elasticsearch-ci/bwc |
| final IngestStats.Stats p1Stats = p1.stats; | ||
| final int ingestCountCompare = Long.compare(p2Stats.ingestCount, p1Stats.ingestCount); | ||
| if (ingestCountCompare == 0) { | ||
| return Long.compare(p2Stats.ingestTimeInMillis, p1Stats.ingestTimeInMillis); |
There was a problem hiding this comment.
I wonder whether ingestTimeInMillis should be the primary sort property?
Because I think now pipelines that are used often but are relatively fast show up before pipelines that are used less often but are slower. I think the latter is a more important signal?
There was a problem hiding this comment.
Yeah I was on the fence about which one to be the primary and which one to be the fallback. I could go either way.
There was a problem hiding this comment.
Maybe @jakelandis would have an opinion here too, since he's looked more into ingest performance than I have
There was a problem hiding this comment.
I prefer ingestTimeInMillis to be the primary sort property in this case, because I think pipelines with higher time spent are more interesting to look at first irregardless of how often these pipelines were used.
There was a problem hiding this comment.
Okay, that works for me too, I'll swap the order here.
dakrone
changed the title
|
@elasticmachine update branch |
dakrone
added
the
auto-merge-without-approval
This commit changes the
/_nodes/statsAPI to return the pipelines in descending order by the timespent executing. If a pipeline ties another pipeline, they are sorted by ingest count.
This ensures that the most used and most expensive pipelines show up first in the JSON response (to
help tracking down which pipelines may be most prevalent).
For example: