Prometheus Receiver is incompatible with Prometheus behaviour on how checks on file_sd_configs are handled

[PaurushGarg]

Component(s)

receiver/prometheus

What happened?

Description

Currently, Prometheus Receiver fails with error during config validation when sd_file is not found but provided in the file_sd_config. However, Prometheus in the absence of sd_files or targetgroups only logs the error and continue to watch the sd_file_path.
This issue was opened on aws-observability/aws-otel-collector repo here.
This issue also breaks the ECS_Observer as given here.

Steps to Reproduce

Run the Otel Collector and Prometheus with the file_sd_config configuration.

Expected Result

Expected to see compatibility between Otel Collector (Prometheus Receiver) and Prometheus handling of the file_sd_config sd_file validation check, and didn't expect Otel Collector to fail in absence of sd_file.

Actual Result

In absence of sd_file, Otel Collector exits, while Prometheus continues to watch the sd_file_path.

Collector version

v0.76.3

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "ecssd"
          file_sd_configs:
            - files:
                - 'ecs_sd_targets.yaml'

exporters:
  logging:
    verbosity: detailed

service:
  telemetry:
    logs:
      level: ${log_level}
  pipelines:
    metrics:
      receivers: [ prometheus ]
      exporters: [ logging ]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[gwiesing]

We are using a AWS EFS where the sd_file is stored. Thus, we can not simply create or point to an empty sd_file during docker build but have to create that file during container start with a shell script to avoid the container fail because of that behaviour.

[PaurushGarg]

Closing this issue as the relevant PR (#21680) of this scope has been merged to opentelemetry-collector-contrib.