Skip to content

Query returns unimplemented error when nesting avg(avg_over_time()) #6202

Closed

Description

Describe the bug
After upgrading from 2.3 to 2.5, this query will result in an unimplemented error:

avg(avg_over_time({job=~"myapps.*"} |= "stats" | json busy="utilization" | unwrap busy [$__interval]))

The utilization field being extracted is a float (ex: 55.4). The avg_over_time() appears to properly calculate the values for each individual stream, but when wrapped with a avg() to get the average across all of the streams the unimplemented error is returned instead.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki v2.5.0
  2. Started Promtail v2.5.0 to tail Kubernetes pods
  3. Query via Grafana v8.4.5: avg(avg_over_time{job="somejob" | json somefield="field" | unwrap somefield [$__interval]))

Expected behavior
While wrapping an avg_over_time() with a avg() is probably not as good as avg_over_time() by (label), there is not an obvious reason why it should not work. I would expect an unwrapped float per log stream could be aggregated with avg().

Environment:

  • Infrastructure: EKS
  • Deployment tool: Raw manifests via ArgoCD

Work-around

Using avg_over_time(...) by (label) to aggregate instead of wrapping it with avg().

This might be related to #6130 and the changes in 2.4, but I haven't had a chance to test that yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions