Use chunked encoding for indices stats response

[original-brownbear]

These responses can become huge, lets chunk them by index.
Unfortunately, slightly awkward to implement this when the chunks depend on the given params but what can you do ...

relates #89838

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I would be inclined to do all the BroadcastResponse implementations at once, changing addCustomXContentFields so it returns a stream of chunks instead. There's only 8 of them. At least we could change addCustomXContentFields to yield a single chunk in each case and then add finer chunking in follow-ups.

[original-brownbear]

I would be inclined to do all the BroadcastResponse implementations at once

Are we sure none of the others generate a large response? The problem of moving to chunked by just wrapping a single chunk like that when the response is large, is that it will mean that we will serialize a large response on the transport thread where it was on whatever other thread it was on before. I'd rather be careful with that tbh.

[DaveCTurner]

Ah good point. Several of them are indeed quite large, and a little too complex to do them all at once I think. Could we introduce a ChunkedBroadcastResponse extends BaseBroadcastResponse implements ChunkedToXContent here since that's where I think we will eventually want to end up and it'll make the other moves a bit more mechanical?

[henningandersen]

LGTM.

[original-brownbear]

here since that's where I think we will eventually want to end up and it'll make the other moves a bit more mechanical?

I just tried this. Currently this would cover IndicesStatsResponse and IndicesSegmentResponse. Both of these work somewhat differently in how they build the response, except for the header field. At least at this point forcing some abstraction onto this doesn't seem to get us much. Maybe try this again when moving the next one and it gets a little clearer what a nice version of this would be?

[DaveCTurner]

I just meant something like this: main...DaveCTurner:elasticsearch:2022-11-24-ChunkedBroadcastResponse-example

i.e. keep the header stuff in the base class and just add the custom stuff in the actual implementations.

[original-brownbear]

Jup that's what I tried too, I guess I didn't really like how it adds extra chunks everywhere but meh maybe that's ok lets do it :)

[original-brownbear]

@DaveCTurner I pushed a775d47 now and used it for the indices stats + existing segment stats.

[DaveCTurner]

LGTM

[henningandersen]

LGTM2

[original-brownbear]

Thanks both!