Description
openedon Nov 12, 2024
Describe the bug
For load shedded requests metrics http_server_requests_seconds*
contain incomplete uri when REST subresource is targetted.
Expected behavior
When triggering URL: http://localhost:8080/hello/sub/in-resource
it is expected that both success and error requests have the same uri in metric:
http_server_requests_seconds_count{method="HEAD",outcome="SUCCESS",status="200",uri="/hello/sub/in-resource",}
http_server_requests_seconds_count{method="HEAD",outcome="SERVER_ERROR",status="503",uri="/hello/sub/in-resource",}
Actual behavior
The actual behavior is the uri is incomplete for error response:
http_server_requests_seconds_count{method="HEAD",outcome="SERVER_ERROR",status="503",uri="/hello/sub",}
How to Reproduce?
quarkus create
quarkus ext add quarkus-micrometer-registry-prometheus
Greetings.java
package org.acme;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
@Path("/hello")
public class GreetingResource {
@GET
@Produces(MediaType.TEXT_PLAIN)
public String hello() {
return "Hello from Quarkus REST";
}
@Path("/sub")
public SubResource getSubResource() {
return new SubResource();
}
}
SubResource.java
package org.acme;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
public class SubResource {
@GET
@Path("/in-resource")
@Produces(MediaType.TEXT_PLAIN)
public String getInSubResource() throws InterruptedException {
Thread.sleep(500);
return "in-resource";
}
}
To simulate load-shedding we need the following:
application.properties
quarkus.thread-pool.queue-size=1
quarkus.thread-pool.max-threads=1
Then execute this command:
curl -I http://localhost:8080/hello/sub/in-resource & \
curl -I http://localhost:8080/hello/sub/in-resource & \
curl -I http://localhost:8080/hello/sub/in-resource
This should cause 1 request to be load shedded:
HTTP/1.1 503 Service Unavailable
And quarkus log should contain:
WARN [io.qua.ver.htt.run.QuarkusErrorHandler] (vert.x-eventloop-thread-1) Worker thread pool exhaustion, no more worker threads available - returning a `503 - SERVICE UNAVAILABLE` response.
Then navigate to http://localhost:8080/q/metrics and observe the difference mentioned in actual and expected behavior.
Output of uname -a
or ver
Darwin mhajas-mac 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:00:32 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6030 arm64
Output of java -version
openjdk version "21.0.4" 2024-07-16 LTS OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode)
Quarkus version or git rev
<quarkus.platform.version>3.16.2</quarkus.platform.version>
Build tool (ie. output of mvnw --version
or gradlew --version
)
Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937) Maven home: /Users/mhajas/.m2/wrapper/dists/apache-maven-3.9.9-bin/33b4b2b4/apache-maven-3.9.9 Java version: 21.0.4, vendor: Eclipse Adoptium, runtime: /Users/mhajas/.sdkman/candidates/java/21.0.4-tem Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "15.1", arch: "aarch64", family: "mac"
Additional information
In VertxHttpServerMetrics
path is obtained from context with key UrlPathTemplate
.
This URL is written there by the ObservabilityHandler
However, the ObservabilityHandler
for subresources is running only when request is processed therefore it is never triggered for load shedded requests.
We encountered this in Keycloak project while creating Grafana dashboard for server errors percentage and error response changes per uri.