Make more proper histogram buckets in PersistenceTimer. (apache#12368)

docs/en/changes/changes.md

@@ -17,6 +17,8 @@
 * Increase `alarm_record#message` column length to 2000 from 200.
 * Remove `alarm_record#message` column indexing.
 * Add Python as a supported language for Pulsar.
+* Make more proper histogram buckets for the `persistence_timer_bulk_prepare_latency`,
+  `persistence_timer_bulk_execute_latency` and `persistence_timer_bulk_all_latency` metrics in PersistenceTimer.
 
 #### UI
 

docs/en/concepts-and-designs/mal.md

@@ -214,6 +214,10 @@ Examples:
 `histogram(le: '<the tag name of le>')`: Transforms less-based histogram buckets to meter system histogram buckets.
 `le` parameter represents the tag name of the bucket.
 
+**Note** In SkyWalking, the histogram buckets are based on time and will be transformed to the `milliseconds-based`
+histogram buckets in the meter system. (If the metrics from the Prometheus are based on the `seconds-based` histogram
+buckets, will multiply the bucket value by 1000.)
+
 #### histogram_percentile
 `histogram_percentile([<p scalar>])`: Represents the meter-system to calculate the p-percentile (0 ≤ p ≤ 100) from the buckets.
 

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/PersistenceTimer.java

@@ -70,16 +70,23 @@ public void start(ModuleManager moduleManager, CoreModuleConfig moduleConfig) {
         prepareLatency = metricsCreator.createHistogramMetric(
             "persistence_timer_bulk_prepare_latency",
             "Latency of the prepare stage in persistence timer",
-            MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE
+            MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE,
+            // 50ms -> 30s should be a proper range for the persistence timer prepare stage
+            .05, .075, .1, .25, .5, .75, 1, 3, 5, 10, 30
         );
         executeLatency = metricsCreator.createHistogramMetric(
             "persistence_timer_bulk_execute_latency",
             "Latency of the execute stage in persistence timer",
-            MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE
+            MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE,
+            // 500ms -> 2min should be a proper range for the persistence timer execute stage
+            // 0.5s, 1s, 3s, 5s, 10s, 15s, 20s, 25s, 50s, 120s, Inf+
+            0.5, 1, 3, 5, 10, 15, 20, 25, 50, 120
         );
         allLatency = metricsCreator.createHistogramMetric(
             "persistence_timer_bulk_all_latency", "Latency of the all stage in persistence timer",
-            MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE
+            MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE,
+            // 500ms -> 2min should be a proper range for the persistence timer
+            0.5, 1, 3, 5, 10, 15, 20, 25, 50, 120
         );
 
         prepareExecutorService = Executors.newFixedThreadPool(moduleConfig.getPrepareThreads());