Could you implement a drill-down feature for service or API traces? When the error rate for a specific service or API is high, clicking on the trend chart should allow users to directly drill down into the list of traces for that time period.

[blanche789]

As shown in the image, when the success rate for a specific API, service, or service instance drops too low, one currently has to manually navigate to the "Trace" section and apply filters. Could a feature be added that allows clicking a specific data point on the trend chart to automatically drill down to the "Trace" page, while simultaneously displaying trace data for a specified time window (e.g., a few minutes before and after) surrounding that point?

[blanche789]

I would also like to ask: currently, configuring alert rules relies on YAML files. Are there any plans to allow defining alert rules via the web interface?

Additionally, which metrics are currently supported for alerts, and is there any documentation available?

[wu-sheng]

We don't have a WIP to add the alarm edit/update via UI. OAP doesn't provide those capabilities, so the new Horizon UI doesn't have magic. We need to discuss this in OAP side first.

[wu-sheng]

As shown in the image, when the success rate for a specific API, service, or service instance drops too low, one currently has to manually navigate to the "Trace" section and apply filters. Could a feature be added that allows clicking a specific data point on the trend chart to automatically drill down to the "Trace" page, while simultaneously displaying trace data for a specified time window (e.g., a few minutes before and after) surrounding that point?

I am not sure how well I can do this. Because only a few metrics and only from the General Service Layer, they are related to native traces.
It has to be a kind of linking structure between metrics and trace query. This is a very new thing.

[wu-sheng]

Implemented — this is up as apache/skywalking-horizon-ui#90.

The key design point (and why it turned out simpler than a general "link metrics to traces" mechanism): only latency and success/error metrics have a natural trace criterion, so the drill is driven by a per-widget mode rather than any metric↔trace mapping table.

How it works — on the General service layer, drill-capable line widgets show a small traces flag in their header and their datapoints are clickable. Click a point and it opens the native Traces list in a new browser tab, pre-filtered to that service (and the selected instance / endpoint) and centered on the clicked bucket's time window:

• latency widgets (response time / percentiles) → slowest-first, with a minimum trace duration of the value you clicked (click the p99 line → the tail; click p50 → median-and-slower).
• error widgets (error rate / success rate / Apdex) → the trace list filtered to ERROR status.

Throughput (cpm) and CPU / memory / GC gauges have no meaningful "which traces" answer, so they offer no drill.

Scope / conditions — a widget gets the drill only when it's a line widget, its metric is latency- or success/error-related, and it's on a layer that serves SkyWalking-native traces (today: the General service layer; Mesh/Zipkin and browser / mini-program use different trace stores). It's a per-widget config field (traceDrill: latency | error) authored in the dashboard widget editor, so it also works on your own custom General-service dashboards, not just the bundled ones.

Widgets enabled out of the box:

Scope	Widget	Mode
Service	Avg Response Time	latency
Service	Response Time Percentile	latency
Service	Error Rate	error
Service	Apdex	error
Instance	Service Instance Latency	latency
Instance	Service Instance Success Rate	error
Endpoint	Response Time	latency
Endpoint	Response Time Percentile	latency
Endpoint	Success Rate	error
Endpoint	MQ Avg Consuming Latency	latency

No backend change was needed — minTraceDuration and traceState already exist on the trace query, so the whole thing is a client-side classify-then-navigate.

[wu-sheng]

Looking at the database dashboard in isolation, there is data.

Read me comments. And you could check the config pages(layer template), virtual database. It uses different metrics(MQE config)

[blanche789]

Regarding the DB success rate, could we also add a trace link for cases where it is below 100%?

[wu-sheng]

Virtual database service is never in Trace, did it?

[blanche789]

Right. However, since database slow logs can be linked to traces, I assumed it would also be possible to link success rate metrics to traces.

[wu-sheng]

That is different, the sampled logs have their associated trace IDs from beginning. There is no web app work.

[blanche789]

Could you clarify the calculation interval for the API failure rate? I’ve configured an alert for the API failure rate, and I’m receiving notifications every 5 minutes; however, the error rate trend chart on the dashboard doesn't show data points at 5-minute intervals—even though the response time chart below does use 5-minute intervals. This inconsistency seems odd.

[wu-sheng]

What is this? I don't know what happened in your side.

[blanche789]

I’ve expanded the time range to two hours. You can see that the response time shows only two data points; however, when I hover the mouse over a specific area, data—including that specific data point—does appear. Why wasn't this existing data point displayed normally? Is this a display bug? When I switch the time range to one hour, the data points display correctly again. I have confirmed that a request is made every 5 minutes and that the metric data exists; however, within certain time ranges, it simply fails to display.

[wu-sheng]

I can't tell what is, even whether it is an issue.
If the traffic doesn't continue in.every minute, it's normal to show like this way.

[blanche789]

It seems like the tag filtering isn't working; could you please take a look and see what the issue is? Packet capture shows that the tag parameter isn't being passed. @wu-sheng

[wu-sheng]

I think you didn't enter? page shows clear that you need that.

[blanche789]

That is indeed the case; I didn't press Enter. Thank you.

[blanche789]

The service metrics in this panel are specific to a particular service rather than the entire system, correct? Was this designed intentionally? @wu-sheng

[wu-sheng]

Good catch — and no, that wasn't intentional. It was a bug on the Overview summary tiles, and it's now fixed.

What was wrong: the Overview KPI tiles (e.g. General services · RPM / Latency / SLA) were aggregating over only a single service instead of the whole layer, so a busy layer read far too low — exactly the "specific to a particular service rather than the entire system" behavior you noticed.

The fix: those tiles now roll up the entire layer — throughput summed, latency and SLA averaged across every service — computed server-side (a layer-wide top_n rollup). So the summary reflects the whole system (per layer) as expected. This covers General, Mesh, and the Virtual Database / Cache / MQ / GenAI layers.

One distinction worth keeping in mind, since the UI has two different "service metrics" surfaces:

• Overview dashboards (the summary tiles) → whole-layer aggregate. This is the part that was wrong and is now fixed.
• Per-layer service dashboards (where you pick a service and see its own charts — the panels the metric→trace drill-down lives on) → scoped to that one service by design. You selected it, and the drill-down opens traces for that service. The system-wide view is the Overview.

Fix is in apache/skywalking-horizon-ui#98.

[wu-sheng]

@blanche789 As this discussion has been very long and has crossed many topics, I am closing this. If you have further questions or bug report, please open an issue or another discussion.

BTW, I will be on family summer vacation, so, away from my laptop until the 29th. I can't fix anything, but available to talk.