[receiver/elasticsearchreceiver]: add support for more metrics

[aboguszewski-sumo]

Is your feature request related to a problem? Please describe.

Sumo Logic app for Elasticsearch uses metrics that are not scraped by the elasticsearchreceiver. The exact list of metrics needed is at the bottom of the post - most of these metrics are the indices metrics.

Additionally, some of the metrics need a host label.

Describe the solution you'd like

New metrics should be added in the metadata.yaml file and have code for them generated. However, this will also require adding modification to the elasticsearchClient interface, as some of the metrics are not being collected right now.

Adding a host label can be resolved similarly as in mysqlreceiver: #14340

Describe alternatives you've considered

No response

Additional context

The missing metrics come from Telegraf plugin for Elasticsearch: link

Similar issues: #14095, #14138

List of needed metrics (disclaimer: I'm still verifying this list and some are already present):

Click to expand

elasticsearch_clusterstats_nodes_jvm_mem_heap_used_in_bytes
elasticsearch_cluster_health_number_of_pending_tasks
elasticsearch_cluster_health_active_primary_shards
elasticsearch_cluster_health_delayed_unassigned_shards
elasticsearch_process_open_file_descriptors
elasticsearch_fs_total_total_in_bytes
elasticsearch_indices_indexing_index_total
elasticsearch_indices_stats_primaries_indexing_index_time_in_millis
elasticsearch_indices_stats_primaries_query_cache_cache_size
elasticsearch_transport_tx_size_in_bytes
elasticsearch_transport_rx_size_in_bytes
elasticsearch_indices_stats_total_search_fetch_time_in_millis
elasticsearch_indices_stats_total_search_query_time_in_millis
elasticsearch_indices_stats_total_indexing_index_time_in_millis
elasticsearch_indices_stats_total_search_query_total
elasticsearch_indices_stats_total_search_fetch_total
elasticsearch_indices_stats_total_refresh_total_time_in_millis
elasticsearch_indices_stats_total_flush_total_time_in_millis
elasticsearch_indices_stats_total_query_cache_evictions
elasticsearch_indices_stats_primaries_query_cache_evictions
elasticsearch_indices_stats_total___fielddata_memory_size_in_bytes
elasticsearch_indices_stats_total_docs_count
elasticsearch_indices_stats_total_indexing_index_total
elasticsearch_indices_stats_total_indexing_delete_total
elasticsearch_indices_stats_total_merges_total_docs
elasticsearch_indices_stats_total_merges_total_size_in_bytes
elasticsearch_indices_stats_total_store_size_in_bytes
elasticsearch_indices_stats_primaries_docs_count
elasticsearch_indices_stats_primaries_segments_index_writer_memory_in_bytes
elasticsearch_indices_stats_total_segments_index_writer_memory_in_bytes
elasticsearch_indices_stats_primaries_segments_count
elasticsearch_indices_stats_total_segments_count
elasticsearch_indices_stats_primaries_segments_memory_in_bytes
elasticsearch_indices_stats_total_segments_memory_in_bytes
elasticsearch_indices_stats_primaries_segments_doc_values_memory_in_bytes
elasticsearch_indices_stats_total_segments_doc_values_memory_in_bytes
elasticsearch_indices_stats_primaries_fielddata_memory_size_in_bytes
elasticsearch_indices_stats_total_fielddata_memory_size_in_bytes
elasticsearch_indices_segments_fixed_bit_set_memory_in_bytes
elasticsearch_indices_stats_total_segments_fixed_bit_set_memory_in_bytes
elasticsearch_indices_segments_terms_memory_in_bytes
elasticsearch_indices_stats_total_segments_terms_memory_in_bytes
elasticsearch_indices_stats_total_translog_operations
elasticsearch_indices_stats_total_translog_size_in_bytes
elasticsearch_indices_stats_total_merges_total_time_in_millis
elasticsearch_os_cpu_load_average_5m
elasticsearch_jvm_mem_heap_used_percent
elasticsearch_thread_pool_get_rejected
elasticsearch_thread_pool_search_queue
elasticsearch_thread_pool_analyze_threads
elasticsearch_thread_pool_analyze_completed
elasticsearch_indices_stats_total___indexing_index_total
elasticsearch_indices_stats_total_refresh_total
elasticsearch_indices_stats_total___merges_total
elasticsearch_indices_get_exists_total
elasticsearch_indices_get_missing_total
elasticsearch_indices_get_exists_time_in_millis
elasticsearch_indices_get_missing_time_in_millis
elasticsearch_indices_get_time_in_millis
elasticsearch_clusterstats_indices_fielddata_evictions

[github-actions]

Pinging code owners: @djaglowski @BinaryFissionGames. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[aboguszewski-sumo]

@djaglowski I've added the pull requests I've been able to do right now. The remaining metrics are:

• more segments metrics - they are blocked by [receiver/elasticsearch]: add segment metrics with total aggregation #14786, but when that one gets merged, they will only have to be emitted
• elasticsearch_jvm_mem_heap_used_percent - technically it can be calculated from existing metrics (jvm.memory.heap.max and jvm.memory.heap.used), so I guess that it wouldn't get merged, would it?
• elasticsearch_clusterstats_indices_fielddata_evictions and elasticsearch_clusterstats_nodes_jvm_mem_heap_used_in_bytes - they are both cluster-level metrics, so I would have to add another endpoint to the client. Technically they can be calculated by fetching these metrics on node level for all nodes and summing them up, but I don't think it's the best idea, what do you think?

[djaglowski]

elasticsearch_jvm_mem_heap_used_percent

We can have this in addition to the max and used metrics. However, it should be disabled by default.

elasticsearch_clusterstats_indices_fielddata_evictions

I think getting it from the API is better, as long as it doesn't add unreasonable load or require much additional config.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/elasticsearch: @djaglowski @BinaryFissionGames

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

@aboguszewski-sumo, it's not clear to me whether this work has been completed. If it has, please close the issue.

[aboguszewski-sumo]

@djaglowski technically this is completed, but I wanted to keep the issue open until the feature gates related to it are removed (which should happen in v0.71).