Improve support of source fallback

[jdconrad]

#88735 adds infrastructure to support source fallback as part of the new scripting fields API. However, grabbing values from source can be quite expensive which leads to the following questions (credit to @lucabelluccini):

1. Should source fallback only be allowed when search.allow_expensive_queries is enabled?
2. Should we have a metric or logging to know if source fallback was used with enough information to support good debugging?
3. Do we appropriately support the new type of sub-object as added by (Add support for dots in field names for metrics usecases #86166)?

Any thoughts on this @romseygeek, @jpountz, and @javanna ?

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[jpountz]

1 is how we used to think but we are more and more departing from it to allow users to do slow things because maybe that's still a better trade-off for them.

Good debugging is important indeed. My expectation is that the field usage stats API should already be able to report when an index performs lots of source access as things are today, but it doesn't tell why and how to address it. One thing we discussed a few times is that field usage stats can be used to figure out which index structures are rarely used and could disabled. But we don't have anything that serves the opposite need of figuring out which index structures would be helpful to add on an index because we had to work around the lack of this index structure by doing something less efficient. I wonder if we could add something like that, that would attribute source access to the fields (including runtime fields) that performed this source access with a counter, so that we could build tooling on top of it that would make recommendations for mapping updates.

[javanna]

Runtime fields already have counts in telemetry around source or doc access (as well as for script-less runtime fields), but these will no longer work once users use the new fields API. I would think it's good info but it's not for debugging purposes. Previously from a script you could already tell whether it accesses source or doc, while now that's transparent hence I think it's a good question, but I am not sure where we would fit that info. Maybe one place could be the profile API.

Do we appropriately support the new type of sub-object as added by (#86166)?

Nothing changes in _source due to the subobjects parameter recently added to mappings. It only affects how objects are dynamically mapped. Leaf fields are stored in the same way in Lucene, as well as left the same way within _source, hence the parameter does not affect loading from doc_values or from _source.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[javanna]

The field api has not introduced anything new when it comes to loading from _source, as that was already possible via runtime fields. We have no current plans around improvements in this area.