Add support for globbing to `resolveCollectorRefs()`

[hhromic]

Is your feature request related to a problem? Please describe.

Currently, the target configuration requires a list of collector names in collectors.
The names must be previously defined via collector_files or inline in the top-level collectors config.

These collector names are resolved from defined collectors using resolveCollectorRefs():

sql_exporter/config/config.go

Lines 546 to 558 in 2ff59aa

	func resolveCollectorRefs(
	collectorRefs []string, collectors map[string]*CollectorConfig, ctx string,
	) ([]*CollectorConfig, error) {
	resolved := make([]*CollectorConfig, 0, len(collectorRefs))
	for _, cref := range collectorRefs {
	c, found := collectors[cref]
	if !found {
	return nil, fmt.Errorf("unknown collector %q referenced in %s", cref, ctx)
	}
	resolved = append(resolved, c)
	}
	return resolved, nil
	}

However, the current implementation is limited only to full collector names to be resolved. When you leverage the handy collector_files config, which supports globbing, it is not sufficient to just add new files that match the globbing pattern. Any new collector names must be added also to target.collectors. Therefore, despite using globbing patterns on the collector file names, one is still forced to keep target.collectors updated manually.

Describe the solution you'd like

A nice enhancement to the target.collectors configuration could be to add support for globbing patterns for the collector names. In this way, it would be possible to easily tell the exporter to "use all collectors with names using these patterns". For example:

global: {}
target:
  data_source_name: driver://host/dbname
  collectors: ['users_*', 'prices_*', 'stocks']
collector_files: ['*.collector.yaml']

For example, a main configuration like this:

global: {}
target:
  data_source_name: driver://host/dbname
  collectors: ['*']
collector_files: ['*.collector.yaml']

Would simply use all available collectors found via collector_files, without altering the same main config file when new collector YAML files are added.

Describe alternatives you've considered

At the moment, we are using a wrapper shell script that scans all the available *.collector.yaml files and extracts the collector_name values from them. Then, we manually generate an array of these names to provide in target.collectors in a generated main config file before starting the SQL exporter.

EDIT: Here is the POSIX sh snippet that collects the collector names from existing files to populate target.collectors:

# collect the names of all available collectors
COLLECTORS=
for COLL_FILE in "$COLLECTORS_DIR"/*.collector.yaml; do
  [ -e "$COLL_FILE" ] || [ -L "$COLL_FILE" ] || continue
  COLLECTOR=$(sed -rn '/^collector_name: / s/collector_name: (.+)/'"'"'\1'"'"'/g p' "$COLL_FILE")
  if [ -z "$COLLECTORS" ]; then
    COLLECTORS=$COLLECTOR
  else
    COLLECTORS="$COLLECTORS, $COLLECTOR"
  fi
done

The above generates a COLLECTORS variable with names surrounded by quotes, ready to use in the YAML file.
As can be seen, having a native solution would be much appreciated to avoid these kind of workarounds.

Additional context

A bit of context: we are using the SQL exporter inside a Docker container, where we want to just provide new collector YAML configuration files without needing to update the main config file inside of the container every time.

[hhromic]

Looks like this can be easily accomplished using Go's built-in Match() function in path/filepath:

• https://pkg.go.dev/path/filepath#Match

If you are interested on this feature request, I could work on it and send a PR. Let me know!

[burningalchemist]

Hey @hhromic, thanks for the provided issues on enhancements and bugs. 👍

For this particular ticket, it looks to be a valid use case, indeed. I appreciate any contributions as I'm pretty limited in time lately. So if you could provide a PR, we could look into it together and help it settle. 🙂

I'm always curious about how people use sql_exporter, organise their configuration files, etc. So many different views, ways of using. 😄

Thanks again! 👍

[hhromic]

Hi @burningalchemist , thanks for being open to contributions!
I will work out the implementation soon and send you a PR for reviewing 👍