Skip to content

http server metrics have wrong uri for load shedded requests #44437

Open

Description

Describe the bug

For load shedded requests metrics http_server_requests_seconds* contain incomplete uri when REST subresource is targetted.

Expected behavior

When triggering URL: http://localhost:8080/hello/sub/in-resource

it is expected that both success and error requests have the same uri in metric:

http_server_requests_seconds_count{method="HEAD",outcome="SUCCESS",status="200",uri="/hello/sub/in-resource",}
http_server_requests_seconds_count{method="HEAD",outcome="SERVER_ERROR",status="503",uri="/hello/sub/in-resource",}

Actual behavior

The actual behavior is the uri is incomplete for error response:

http_server_requests_seconds_count{method="HEAD",outcome="SERVER_ERROR",status="503",uri="/hello/sub",}

How to Reproduce?

quarkus create
quarkus ext add quarkus-micrometer-registry-prometheus

Greetings.java

package org.acme;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;


@Path("/hello")
public class GreetingResource {
    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello() {
        return "Hello from Quarkus REST";
    }

    @Path("/sub")
    public  SubResource getSubResource() {
        return new SubResource();
    }
}

SubResource.java

package org.acme;

import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

public class SubResource {

    @GET
    @Path("/in-resource")
    @Produces(MediaType.TEXT_PLAIN)
    public String getInSubResource() throws InterruptedException {
        Thread.sleep(500);
        return "in-resource";
    }
}

To simulate load-shedding we need the following:
application.properties

quarkus.thread-pool.queue-size=1
quarkus.thread-pool.max-threads=1

Then execute this command:

curl -I http://localhost:8080/hello/sub/in-resource & \
curl -I http://localhost:8080/hello/sub/in-resource & \
curl -I http://localhost:8080/hello/sub/in-resource

This should cause 1 request to be load shedded:

HTTP/1.1 503 Service Unavailable

And quarkus log should contain:

WARN  [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-1) Worker thread pool exhaustion, no more worker threads available - returning a `503 - SERVICE UNAVAILABLE` response.

Then navigate to http://localhost:8080/q/metrics and observe the difference mentioned in actual and expected behavior.

Output of uname -a or ver

Darwin mhajas-mac 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:00:32 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6030 arm64

Output of java -version

openjdk version "21.0.4" 2024-07-16 LTS OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)

Quarkus version or git rev

<quarkus.platform.version>3.16.2</quarkus.platform.version>

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937) Maven home: /Users/mhajas/.m2/wrapper/dists/apache-maven-3.9.9-bin/33b4b2b4/apache-maven-3.9.9 Java version: 21.0.4, vendor: Eclipse Adoptium, runtime: /Users/mhajas/.sdkman/candidates/java/21.0.4-tem Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "15.1", arch: "aarch64", family: "mac"

Additional information

In VertxHttpServerMetrics path is obtained from context with key UrlPathTemplate.

This URL is written there by the ObservabilityHandler

However, the ObservabilityHandler for subresources is running only when request is processed therefore it is never triggered for load shedded requests.

We encountered this in Keycloak project while creating Grafana dashboard for server errors percentage and error response changes per uri.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions