Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define SemConv predefined JQ functions #246

Merged
merged 41 commits into from
Jul 22, 2024

Conversation

lquerel
Copy link
Contributor

@lquerel lquerel commented Jul 17, 2024

This PR adds a set of predefined JQ functions to simplify the processing and generation of semconv documentation/code.

Process Registry Attributes

The following JQ filter extracts the registry attributes from the resolved registry and returns a list of registry attributes grouped by namespace and sorted by attribute names.

templates:
  - pattern: attributes.j2
    filter: semconv_grouped_attributes
    application_mode: each

The output of the JQ filter has the following structure:

[
  {
    "root_namespace": "user_agent",
    "attributes": [
      {
        "brief": "Value of the HTTP User-Agent",
        "examples": [ ... ],
        "name": "user_agent.original",
        "root_namespace": "user_agent",
        "requirement_level": "recommended",
        "stability": "stable",
        "type": "string",
        // ... other fields
      }, 
      // ... other attributes in the same root namespace
    ]
  },
  // ... other root namespaces
]

The semconv_grouped_attributes function also supports options to exclude specified root namespaces, specific stability levels, and deprecated entities. The following syntax is supported:

templates:
  - pattern: attributes.j2
    filter: >
      semconv_grouped_attributes({
        "exclude_root_namespace": ["url", "network"], 
        "exclude_stability": ["experimental"],
        "exclude_deprecated": true,
      })
    application_mode: each

The structure of the output of semconv_grouped_attributes with these options is exactly the same as without the options. The JSON object passed as a parameter describes a series of options that can easily be extended if needed. Each of these options is optional. It's also possible to use the special parameter $params as follows:

params:
  exclude_root_namespace: ["url", "network"] 
  exclude_stability: ["experimental"]
  exclude_deprecated: true

templates:
  - pattern: attributes.j2
    filter: semconv_grouped_attributes($params)
    application_mode: each

Technically, the semconv_grouped_attributes function is a combination of two semconv JQ functions:

def semconv_grouped_attributes($options):
    semconv_attributes($options)
    | semconv_group_attributes_by_root_namespace;

def semconv_grouped_attributes: semconv_grouped_attributes({});

The semconv_attributes function extracts the registry attributes and applies the given options. The semconv_group_attributes_by_root_namespace function groups the attributes by root namespace. It's possible to combine these two functions with your own JQ filters if needed.

Process Metrics

The following JQ filter extracts the metrics from the resolved registry, sorted by group root namespace and sorted by metric names.

templates:
  - pattern: metrics.j2
    filter: semconv_grouped_metrics
    application_mode: each

The output of the JQ filter has the following structure:

[
  {
    "root_namespace": "jvm",
    "metrics": [
      {
        "attributes": [ ... ],
        "brief": "Recent CPU utilization for the process as reported by the JVM.",
        "id": "metric.jvm.cpu.recent_utilization",
        "instrument": "gauge",
        "metric_name": "jvm.cpu.recent_utilization",
        "root_namespace": "jvm",
        "note": "The value range is [0.0,1.0]. ...",
        "stability": "stable",
        "type": "metric",
        "unit": "1",
        // ... other fields
      },
      // ... other metrics in the same root namespace
    ]
  },
  // ... other root namespaces
]

The same options are supported by semconv_grouped_metrics, as shown in the following example:

params:
  exclude_root_namespace: ["url", "network"] 
  exclude_stability: ["experimental"]
  exclude_deprecated: true

templates:
  - pattern: metrics.j2
    filter: semconv_grouped_metrics($params)
    application_mode: each

All the semconv_grouped_<...> functions are the composition of two functions: semconv_<...> and semconv_group_<...>_by_root_namespace.

Other improvements

This PR contains an update of the Weaver Forge README.md.
This PR updates the Rust example to leverage both the whitespace control settings and the new JQ filters.

lquerel and others added 24 commits July 5, 2024 21:51
# Conflicts:
#	crates/weaver_forge/src/config.rs
#	crates/weaver_forge/src/extensions/case.rs
#	crates/weaver_forge/src/lib.rs
@lquerel lquerel self-assigned this Jul 17, 2024
@lquerel lquerel added the enhancement New feature or request label Jul 17, 2024
@lquerel lquerel linked an issue Jul 17, 2024 that may be closed by this pull request
crates/weaver_forge/src/lib.rs Outdated Show resolved Hide resolved
@lquerel lquerel marked this pull request as ready for review July 20, 2024 00:31
@lquerel lquerel requested a review from jsuereth as a code owner July 20, 2024 00:31
@lquerel lquerel changed the title [WIP] Define SemConv predefined JQ functions Define SemConv predefined JQ functions Jul 20, 2024
@lquerel lquerel requested a review from lmolkova July 22, 2024 14:45
Copy link
Contributor

@jsuereth jsuereth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Just called out some of my concerns going forward on formalizing semconv decisions over here in the weaver codebase.

defaults/jq/semconv.jq Show resolved Hide resolved
defaults/jq/semconv.jq Outdated Show resolved Hide resolved
@lmolkova
Copy link
Contributor

Could we postpone adding semconv_grouped_resources, semconv_grouped_scopes, semconv_grouped_spans, semconv_grouped_events until we have a use-case?

Copy link
Contributor

@lmolkova lmolkova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some minor comments, looks great otherwise!

defaults/jq/semconv.jq Show resolved Hide resolved
defaults/jq/semconv.jq Outdated Show resolved Hide resolved
@lquerel
Copy link
Contributor Author

lquerel commented Jul 22, 2024

@lmolkova, @jsuereth, once I get confirmation for the few questions I have (see previous comment), I will make the changes, bump the weaver version, and create a new release. This release will be a good starting point for the codegen authors as these new JQ filters improve the UX. The step-by-step guide should also ease the process of the codegen migration.

@lquerel lquerel merged commit 5661695 into open-telemetry:main Jul 22, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
3 participants