agent ignores 'path_prefix'

[kevinschoonover]

Describe the bug

Agent does not prepend the 'profiling_config' argument when scraping targets

scrape_configs:
- job_name: vault
  authorization:
    credentials_file: "/secrets/vault_token"
  scheme: https
  profiling_config:
    path_prefix: '/v1/sys/pprof'
  tls_config:
    insecure_skip_verify: true
  static_configs:
    - targets:
        - 100.114.243.60:8200
        - 100.93.8.11:8200
        - 100.111.89.83:8200
        

      labels:
        env: prod

evel=error caller=target.go:183 ts=2022-11-06T08:58:19.92511409Z msg="fetch profile failed" target="{__address__=\"100.111.89.83:8200\", __name__=\"process_cpu\", __profile_path__=\"/debug/pprof/profile\", __scheme__=\"https\", env=\"prod\", instance=\"100.111.89.83:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"
level=error caller=target.go:183 ts=2022-11-06T08:58:22.352563943Z msg="fetch profile failed" target="{__address__=\"100.93.8.11:8200\", __name__=\"block\", __profile_path__=\"/debug/pprof/block\", __scheme__=\"https\", env=\"prod\", instance=\"100.93.8.11:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"
level=error caller=target.go:183 ts=2022-11-06T08:58:22.770612925Z msg="fetch profile failed" target="{__address__=\"100.114.243.60:8200\", __name__=\"mutex\", __profile_path__=\"/debug/pprof/mutex\", __scheme__=\"https\", env=\"prod\", instance=\"100.114.243.60:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"
level=error caller=target.go:183 ts=2022-11-06T08:58:23.198404024Z msg="fetch profile failed" target="{__address__=\"100.93.8.11:8200\", __name__=\"mutex\", __profile_path__=\"/debug/pprof/mutex\", __scheme__=\"https\", env=\"prod\", instance=\"100.93.8.11:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"
level=error caller=target.go:183 ts=2022-11-06T08:58:23.201921427Z msg="fetch profile failed" target="{__address__=\"100.93.8.11:8200\", __name__=\"goroutine\", __profile_path__=\"/debug/pprof/goroutine\", __scheme__=\"https\", env=\"prod\", instance=\"100.93.8.11:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"
level=error caller=target.go:183 ts=2022-11-06T08:58:23.296266503Z msg="fetch profile failed" target="{__address__=\"100.111.89.83:8200\", __name__=\"goroutine\", __profile_path__=\"/debug/pprof/goroutine\", __scheme__=\"https\", env=\"prod\", instance=\"100.111.89.83:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"
level=error caller=target.go:183 ts=2022-11-06T08:58:24.253265484Z msg="fetch profile failed" target="{__address__=\"100.111.89.83:8200\", __name__=\"block\", __profile_path__=\"/debug/pprof/block\", __scheme__=\"https\", env=\"prod\", instance=\"100.111.89.83:8200\", job=\"vault\"}" err="server returned HTTP status (404) {\"errors\":[]}"

To Reproduce

Steps to reproduce the behavior:

1. Start Phlare 0.1.0 with the scrape configuration above and look at the error messages by the agent

Environment

• Infrastructure: Nomad
• Deployment tool: [e.g., helm, jsonnet]

Additional Context

I see you all are planning on driving this feature forward with grafana-agent in grafana/pyroscope#2101 so happy to close this as a 'work fix, please wait for grafana agent' since I can hard cord all of the paths in the pprof_config

[cyriltovena]

Found a bug that causes this. On it.

[cyriltovena]

Related to parca-dev/parca@2182734

[kawhicurry]

Meet the same problem. I fix it by specify path manually.