Prepare Prometheus alerting rules for Istio #65

bzurkowski · 2020-04-26T17:29:05Z

Istio collects a rich set of metrics related to the data plane and control plane. These metrics are stored in Prometheus.

We could leverage the collected metrics to produce alerts describing service mesh performance, for instance:

high error rate on the communication path between two services,
high latency on the communication path between two services,
failure on configuring a proxy by Istio Pliot,
failure on rotating a proxy certificate by Istio Citadel.
failure on validating a config by Istio Galley.

First, we should look for existing alerting rules in open source and evaluate them. Second, we should attempt to elaborate custom set of alerting rules.

bzurkowski · 2020-04-26T17:32:19Z

Sample entry for service-level metric request_total_count:

istio_requests_total{
  connection_security_policy="mutual_tls",
  destination_app="details",
  destination_canonical_service="details",
  destination_canonical_revision="v1",
  destination_principal="cluster.local/ns/default/sa/default",
  destination_service="details.default.svc.cluster.local",
  destination_service_name="details",
  destination_service_namespace="default",
  destination_version="v1",
  destination_workload="details-v1",
  destination_workload_namespace="default",
  reporter="destination",
  request_protocol="http",
  response_code="200",
  response_flags="-",
  source_app="productpage",
  source_canonical_service="productpage",
  source_canonical_revision="v1",
  source_principal="cluster.local/ns/default/sa/default",
  source_version="v1",
  source_workload="productpage-v1",
  source_workload_namespace="default"
} 214

bzurkowski · 2020-04-26T17:35:31Z

Metrics exported by Istio Pilot.
Metrics exported by Istio Galley.
Metrics exported by Istio Citadel.
Metrics exported by Istio Mixer.
Metrics exported by Istio Sidecar Injector.

bzurkowski · 2020-04-26T17:40:46Z

According to initial research, alerting rules for Istio are a rare treasure:

Awesome Prometheus rules - alerting rules missing.
Discussion in Istio community - alerting rules not provided by Istio community.

bzurkowski · 2020-08-07T10:25:37Z

Let's implement the item according to the following plan:

Search for alerting rules available in open source.
Deploy Istio and sample app in the testbed.
If possible, evaluate alerting rules found in open source.
Collect and analyze metrics generated by the service mesh (Envoy, Istio control plane).
Prepare a proposal of alerts based on collected metrics.
Implement alerting rules.
Test alerting rules in the testbed.

aleksandra-galara · 2020-08-26T06:56:51Z

As for now, the only alerts in open source, which I was able to find, are the ones below, so I've collected Prometheus metrics from Istio and I'll be working on creating our own alerts.

 - alert: IstioPilotAvailabilityDrop
    annotations:
      summary: 'Istio Pilot Availability Drop'
      description: 'Pilot pods have dropped during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Envoy sidecars might have outdated configuration'
    expr: >
      avg(avg_over_time(up{job="pilot"}[1m])) < 0.5
    for: 5m

  - alert: IstioMixerTelemetryAvailabilityDrop
    annotations:
      summary: 'Istio Mixer Telemetry Drop'
      description: 'Mixer pods have dropped during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Istio metrics will not work correctly'
    expr: >
      avg(avg_over_time(up{job="mixer", service="istio-telemetry", endpoint="http-monitoring"}[5m])) < 0.5
    for: 5m

  - alert: IstioGalleyAvailabilityDrop
    annotations:
      summary: 'Istio Galley Availability Drop'
      description: 'Galley pods have dropped during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Istio config ingestion and processing will not work'
    expr: >
      avg(avg_over_time(up{job="galley"}[5m])) < 0.5
    for: 5m

  - alert: IstioGatewayAvailabilityDrop
    annotations:
      summary: 'Istio Gateway Availability Drop'
      description: 'Gateway pods have dropped during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Inbound traffic will likely be affected'
    expr: >
      min(kube_deployment_status_replicas_available{deployment="istio-ingressgateway", namespace="istio-system"}) without (instance, pod) < 2
    for: 5m

  - alert: IstioPilotPushErrorsHigh
    annotations:
      summary: 'Number of Istio Pilot push errors is too high'
      description: 'Pilot has too many push errors during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Envoy sidecars might have outdated configuration'
    expr: >
      sum(irate(pilot_xds_push_errors{job="pilot"}[5m])) / sum(irate(pilot_xds_pushes{job="pilot"}[5m])) > 0.05
    for: 5m

  - alert: IstioMixerPrometheusDispatchesLow
    annotations:
      summary: 'Number of Mixer dispatches to Prometheus is too low'
      description: 'Mixer disptaches to Prometheus has dropped below normal levels during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Istio metrics might not be being exported properly'
    expr: >
      sum(irate(mixer_runtime_dispatches_total{adapter=~"prometheus"}[5m])) < 180
    for: 5m

  - alert: IstioGlobalRequestRateHigh
    annotations:
      summary: 'Istio Global Request Rate High'
      description: 'Istio global request rate is unusually high during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). The amount of traffic being generated inside the service mesh is higher than normal'
    expr: >
      round(sum(irate(istio_requests_total{reporter="destination"}[5m])), 0.001) > 1200
    for: 5m

  - alert: IstioGlobalRequestRateLow
    annotations:
      summary: 'Istio global request rate too low'
      description: 'Istio global request rate is unusually low during the last 5m (current value: *{{ printf "%2.0f%%" $value }}*). The amount of traffic being generated inside the service mesh has dropped below usual levels'
    expr: >
      round(sum(irate(istio_requests_total{reporter="destination"}[5m])), 0.001) < 300
    for: 5m

  - alert: IstioGlobalHTTP5xxRateHigh
    annotations:
      summary: 'Istio Percentage of HTTP 5xx responses is too high'
      description: 'Istio global HTTP 5xx rate is too high in last 5m (current value: *{{ printf "%2.0f%%" $value }}*). The HTTP 5xx errors within the service mesh is unusually high'
    expr: >
       sum(irate(istio_requests_total{reporter="destination", response_code=~"5.*"}[5m])) / sum(irate(istio_requests_total{reporter="destination"}[5m])) > 0.01
    for: 5m

  - alert: IstioGatewayOutgoingSuccessLow
    annotations:
      summary: 'Istio Gateway outgoing success rate is too low'
      description: 'Istio Gateway success to outbound destinations is too low in last 5m (current value: *{{ printf "%2.0f%%" $value }}*). Inbound traffic may be affected'
    expr: >
      sum(irate(istio_requests_total{reporter="source", source_workload="istio-ingressgateway",source_workload_namespace="istio-system", connection_security_policy!="mutual_tls",response_code!~"5.*"}[5m])) /  sum(irate(istio_requests_total{reporter="source", source_workload="istio-ingressgateway",source_workload_namespace="istio-system", connection_security_policy!="mutual_tls"}[5m])) < 0.995
    for: 5m

aleksandra-galara · 2020-09-03T07:17:01Z

Collected Pilot metrics:

# HELP citadel_server_csr_count The number of CSRs received by Citadel server.
# TYPE citadel_server_csr_count counter
citadel_server_csr_count 18
# HELP citadel_server_root_cert_expiry_timestamp The unix timestamp, in seconds, when Citadel root cert will expire. We set it to negative in case of internal error.
# TYPE citadel_server_root_cert_expiry_timestamp gauge
citadel_server_root_cert_expiry_timestamp 1.913711108e+09
# HELP citadel_server_success_cert_issuance_count The number of certificates issuances that have succeeded.
# TYPE citadel_server_success_cert_issuance_count counter
citadel_server_success_cert_issuance_count 18
# HELP endpoint_no_pod Endpoints without an associated pod.
# TYPE endpoint_no_pod gauge
endpoint_no_pod 0
# HELP galley_validation_cert_key_updates Galley validation webhook certificate updates
# TYPE galley_validation_cert_key_updates counter
galley_validation_cert_key_updates 1
# HELP galley_validation_config_update_error k8s webhook configuration update error
# TYPE galley_validation_config_update_error counter
galley_validation_config_update_error{reason="NotFound"} 2
# HELP galley_validation_config_updates k8s webhook configuration updates
# TYPE galley_validation_config_updates counter
galley_validation_config_updates 2
# HELP galley_validation_failed Resource validation failed
# TYPE galley_validation_failed counter
galley_validation_failed{group="networking.istio.io",reason="invalid_resource",resource="gateways",version="v1alpha3"} 1
# HELP galley_validation_passed Resource is valid
# TYPE galley_validation_passed counter
galley_validation_passed{group="networking.istio.io",resource="gateways",version="v1alpha3"} 1
galley_validation_passed{group="networking.istio.io",resource="virtualservices",version="v1alpha3"} 1
galley_validation_passed{group="security.istio.io",resource="peerauthentications",version="v1beta1"} 1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.09e-06
go_gc_duration_seconds{quantile="0.25"} 1.6072e-05
go_gc_duration_seconds{quantile="0.5"} 2.1731e-05
go_gc_duration_seconds{quantile="0.75"} 3.3011e-05
go_gc_duration_seconds{quantile="1"} 0.000401636
go_gc_duration_seconds_sum 0.116546106
go_gc_duration_seconds_count 687
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 396
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.2"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.3478712e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.2156622216e+10
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 2.754884e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.65354905e+08
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 1.2193568177902437e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.840264e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.3478712e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 2.3445504e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.9993344e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 256649
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 1.6351232e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.3438848e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.598427877629561e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.65611554e+08
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 535024
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 688128
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.9452384e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.89202e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 3.670016e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 3.670016e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.6300544e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 17
# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 18
grpc_server_handled_total{grpc_code="OK",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
# HELP grpc_server_handling_seconds Histogram of response latency (seconds) of gRPC that had been application-level handled by the server.
# TYPE grpc_server_handling_seconds histogram
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.005"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.01"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.025"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.05"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.1"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.25"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="0.5"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="1"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="2.5"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="5"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="10"} 18
grpc_server_handling_seconds_bucket{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary",le="+Inf"} 18
grpc_server_handling_seconds_sum{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 0.058094681
grpc_server_handling_seconds_count{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 18
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.005"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.01"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.025"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.05"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.1"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.25"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.5"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="1"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="2.5"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="5"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="10"} 0
grpc_server_handling_seconds_bucket{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="+Inf"} 0
grpc_server_handling_seconds_sum{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handling_seconds_count{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.005"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.01"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.025"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.05"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.1"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.25"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="0.5"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="1"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="2.5"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="5"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="10"} 0
grpc_server_handling_seconds_bucket{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary",le="+Inf"} 0
grpc_server_handling_seconds_sum{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handling_seconds_count{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.005"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.01"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.025"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.05"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.1"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.25"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="0.5"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="1"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="2.5"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="5"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="10"} 0
grpc_server_handling_seconds_bucket{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream",le="+Inf"} 0
grpc_server_handling_seconds_sum{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_handling_seconds_count{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
# HELP grpc_server_msg_received_total Total number of RPC stream messages received on the server.
# TYPE grpc_server_msg_received_total counter
grpc_server_msg_received_total{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 18
grpc_server_msg_received_total{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_msg_received_total{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
# HELP grpc_server_msg_sent_total Total number of gRPC stream messages sent by the server.
# TYPE grpc_server_msg_sent_total counter
grpc_server_msg_sent_total{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 18
grpc_server_msg_sent_total{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_msg_sent_total{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="CreateCertificate",grpc_service="istio.v1.auth.IstioCertificateService",grpc_type="unary"} 18
grpc_server_started_total{grpc_method="DeltaAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
grpc_server_started_total{grpc_method="HandleCSR",grpc_service="istio.v1.auth.IstioCAService",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="StreamAggregatedResources",grpc_service="envoy.service.discovery.v2.AggregatedDiscoveryService",grpc_type="bidi_stream"} 0
# HELP istio_build Istio component build info
# TYPE istio_build gauge
istio_build{component="pilot",tag="1.6.8"} 1
# HELP pilot_conflict_inbound_listener Number of conflicting inbound listeners.
# TYPE pilot_conflict_inbound_listener gauge
pilot_conflict_inbound_listener 0
# HELP pilot_conflict_outbound_listener_http_over_current_tcp Number of conflicting wildcard http listeners with current wildcard tcp listener.
# TYPE pilot_conflict_outbound_listener_http_over_current_tcp gauge
pilot_conflict_outbound_listener_http_over_current_tcp 0
# HELP pilot_conflict_outbound_listener_tcp_over_current_http Number of conflicting wildcard tcp listeners with current wildcard http listener.
# TYPE pilot_conflict_outbound_listener_tcp_over_current_http gauge
pilot_conflict_outbound_listener_tcp_over_current_http 0
# HELP pilot_conflict_outbound_listener_tcp_over_current_tcp Number of conflicting tcp listeners with current tcp listener.
# TYPE pilot_conflict_outbound_listener_tcp_over_current_tcp gauge
pilot_conflict_outbound_listener_tcp_over_current_tcp 0
# HELP pilot_destrule_subsets Duplicate subsets across destination rules for same host
# TYPE pilot_destrule_subsets gauge
pilot_destrule_subsets 0
# HELP pilot_duplicate_envoy_clusters Duplicate envoy clusters caused by service entries with same hostname
# TYPE pilot_duplicate_envoy_clusters gauge
pilot_duplicate_envoy_clusters 0
# HELP pilot_eds_no_instances Number of clusters without instances.
# TYPE pilot_eds_no_instances gauge
pilot_eds_no_instances 0
# HELP pilot_endpoint_not_ready Endpoint found in unready state.
# TYPE pilot_endpoint_not_ready gauge
pilot_endpoint_not_ready 0
# HELP pilot_inbound_updates Total number of updates received by pilot.
# TYPE pilot_inbound_updates counter
pilot_inbound_updates{type="config"} 150
pilot_inbound_updates{type="eds"} 81
pilot_inbound_updates{type="svc"} 62
pilot_inbound_updates{type="svcdelete"} 1
# HELP pilot_info Pilot version and build information.
# TYPE pilot_info gauge
pilot_info{version="1.6.8-5bf4ade4e4ba40977a1bcacc07740631fb56a099-dirty-Modified"} 1
# HELP pilot_k8s_cfg_events Events from k8s config.
# TYPE pilot_k8s_cfg_events counter
pilot_k8s_cfg_events{event="add",type="EnvoyFilter"} 10
pilot_k8s_cfg_events{event="add",type="Gateway"} 2
pilot_k8s_cfg_events{event="add",type="PeerAuthentication"} 1
pilot_k8s_cfg_events{event="add",type="ServiceEntry"} 2
pilot_k8s_cfg_events{event="add",type="VirtualService"} 3
pilot_k8s_cfg_events{event="updatesame",type="EnvoyFilter"} 12800
pilot_k8s_cfg_events{event="updatesame",type="Gateway"} 2479
pilot_k8s_cfg_events{event="updatesame",type="PeerAuthentication"} 1280
pilot_k8s_cfg_events{event="updatesame",type="ServiceEntry"} 2560
pilot_k8s_cfg_events{event="updatesame",type="VirtualService"} 3759
# HELP pilot_k8s_reg_events Events from k8s registry.
# TYPE pilot_k8s_reg_events counter
pilot_k8s_reg_events{event="add",type="Endpoints"} 65
pilot_k8s_reg_events{event="add",type="Nodes"} 3
pilot_k8s_reg_events{event="add",type="Pods"} 90
pilot_k8s_reg_events{event="add",type="Services"} 62
pilot_k8s_reg_events{event="delete",type="Endpoints"} 1
pilot_k8s_reg_events{event="delete",type="Pods"} 1
pilot_k8s_reg_events{event="delete",type="Services"} 1
pilot_k8s_reg_events{event="update",type="Endpoints"} 34
pilot_k8s_reg_events{event="update",type="Nodes"} 23014
pilot_k8s_reg_events{event="update",type="Pods"} 61
pilot_k8s_reg_events{event="updatesame",type="Endpoints"} 196680
pilot_k8s_reg_events{event="updatesame",type="Nodes"} 3840
pilot_k8s_reg_events{event="updatesame",type="Pods"} 113992
pilot_k8s_reg_events{event="updatesame",type="Services"} 78157
# HELP pilot_no_ip Pods not found in the endpoint table, possibly invalid.
# TYPE pilot_no_ip gauge
pilot_no_ip 0
# HELP pilot_proxy_convergence_time Delay in seconds between config change and a proxy receiving all required configuration.
# TYPE pilot_proxy_convergence_time histogram
pilot_proxy_convergence_time_bucket{le="0.1"} 57
pilot_proxy_convergence_time_bucket{le="0.5"} 63
pilot_proxy_convergence_time_bucket{le="1"} 63
pilot_proxy_convergence_time_bucket{le="3"} 63
pilot_proxy_convergence_time_bucket{le="5"} 63
pilot_proxy_convergence_time_bucket{le="10"} 63
pilot_proxy_convergence_time_bucket{le="20"} 63
pilot_proxy_convergence_time_bucket{le="30"} 63
pilot_proxy_convergence_time_bucket{le="+Inf"} 63
pilot_proxy_convergence_time_sum 2.022021797
pilot_proxy_convergence_time_count 63
# HELP pilot_proxy_queue_time Time in seconds, a proxy is in the push queue before being dequeued.
# TYPE pilot_proxy_queue_time histogram
pilot_proxy_queue_time_bucket{le="0.1"} 95
pilot_proxy_queue_time_bucket{le="1"} 95
pilot_proxy_queue_time_bucket{le="3"} 95
pilot_proxy_queue_time_bucket{le="5"} 95
pilot_proxy_queue_time_bucket{le="10"} 95
pilot_proxy_queue_time_bucket{le="20"} 95
pilot_proxy_queue_time_bucket{le="30"} 95
pilot_proxy_queue_time_bucket{le="+Inf"} 95
pilot_proxy_queue_time_sum 0.0021188289999999996
pilot_proxy_queue_time_count 95
# HELP pilot_push_triggers Total number of times a push was triggered, labeled by reason for the push.
# TYPE pilot_push_triggers counter
pilot_push_triggers{type="config"} 18
pilot_push_triggers{type="endpoint"} 77
pilot_push_triggers{type="service"} 21
# HELP pilot_rds_expired_nonce Total number of RDS messages with an expired nonce.
# TYPE pilot_rds_expired_nonce counter
pilot_rds_expired_nonce 1
# HELP pilot_services Total services known to pilot.
# TYPE pilot_services gauge
pilot_services 64
# HELP pilot_virt_services Total virtual services known to pilot.
# TYPE pilot_virt_services gauge
pilot_virt_services 3
# HELP pilot_vservice_dup_domain Virtual services with dup domains.
# TYPE pilot_vservice_dup_domain gauge
pilot_vservice_dup_domain 0
# HELP pilot_xds Number of endpoints connected to this pilot using XDS.
# TYPE pilot_xds gauge
pilot_xds{version="1.6.8"} 9
# HELP pilot_xds_eds_all_locality_endpoints Network endpoints for each cluster(across all localities), as of last push. Zero endpoints is an error.
# TYPE pilot_xds_eds_all_locality_endpoints gauge
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.10050_._.zabbix-agent.zabbix.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.10051_._.zabbix-proxy-mysql.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.10051_._.zabbix-proxy-sqlite3.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.10051_._.zabbix-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.10052_._.zabbix-java-gateway.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.14250_._.jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.14267_._.jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.14268_._.jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.15010_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.15012_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.15014_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.15021_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.15443_._.istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.15443_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.16686_._.jaeger-query.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.20001_._.kiali.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.3000_._.grafana.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.31400_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.3306_._.mysql-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.3550_._.productcatalogservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.44134_._.tiller-deploy.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.kibana.logging.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.kubernetes-dashboard.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.kubernetes.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.metrics-server.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.443_._.zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.5000_._.emailservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.5000_._.orca.rca.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.5000_._.registry.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.50051_._.paymentservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.50051_._.shippingservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.5050_._.checkoutservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.53_._.coredns.kube-system.svc.cluster.local"} 2
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.6379_._.redis-cart.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.7000_._.currencyservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.7070_._.cartservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.8080_._.prometheus-kube-state-metrics.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.8080_._.prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.8080_._.recommendationservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.8080_._.web.default.svc.cluster.local"} 0
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.frontend-external.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.frontend.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.prometheus-grafana.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.tracing.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.80_._.zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.8529_._.arangodb.rca.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.853_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.8765_._.falco.falco.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9080_._.details.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9080_._.productpage.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9080_._.ratings.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9080_._.reviews.default.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9090_._.prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9090_._.prometheus.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9093_._.prometheus-prometheus-oper-alertmanager.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9100_._.prometheus-prometheus-node-exporter.monitoring.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9153_._.coredns.kube-system.svc.cluster.local"} 2
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9200_._.elasticsearch-client.logging.svc.cluster.local"} 2
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9411_._.zipkin.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound_.9555_._.adservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|10050||zabbix-agent.zabbix.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound|10051||zabbix-proxy-mysql.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|10051||zabbix-proxy-sqlite3.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|10051||zabbix-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|10052||zabbix-java-gateway.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|14250||jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|14267||jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|14268||jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|15010||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|15012||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|15014||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|15021||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|15443||istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|16686||jaeger-query.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|20001||kiali.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|3000||grafana.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|3306||mysql-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|3550||productcatalogservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|44134||tiller-deploy.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||kibana.logging.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||kubernetes-dashboard.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||kubernetes.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||metrics-server.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|443||zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|5000||emailservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|5000||orca.rca.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|5000||registry.kube-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|50051||paymentservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|50051||shippingservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|5050||checkoutservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|53||coredns.kube-system.svc.cluster.local"} 2
pilot_xds_eds_all_locality_endpoints{cluster="outbound|6379||redis-cart.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|7000||currencyservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|7070||cartservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|8080||prometheus-kube-state-metrics.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|8080||prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|8080||recommendationservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|8080||web.default.svc.cluster.local"} 0
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||frontend-external.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||frontend.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||prometheus-grafana.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||tracing.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|80||zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|8529||arangodb.rca.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|853||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|8765||falco.falco.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9080||details.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9080||productpage.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9080||ratings.default.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9080||reviews.default.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9090||prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9090||prometheus.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9093||prometheus-prometheus-oper-alertmanager.monitoring.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9100||prometheus-prometheus-node-exporter.monitoring.svc.cluster.local"} 3
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9153||coredns.kube-system.svc.cluster.local"} 2
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9200||elasticsearch-client.logging.svc.cluster.local"} 2
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9411||zipkin.istio-system.svc.cluster.local"} 1
pilot_xds_eds_all_locality_endpoints{cluster="outbound|9555||adservice.hipster-shop.svc.cluster.local"} 1
# HELP pilot_xds_eds_instances Instances for each cluster(grouped by locality), as of last push. Zero instances is an error.
# TYPE pilot_xds_eds_instances gauge
pilot_xds_eds_instances{cluster="outbound_.10050_._.zabbix-agent.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.10051_._.zabbix-proxy-mysql.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.10051_._.zabbix-proxy-sqlite3.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.10051_._.zabbix-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.10052_._.zabbix-java-gateway.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.14250_._.jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.14267_._.jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.14268_._.jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.15010_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.15012_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.15014_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.15021_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.15443_._.istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.15443_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.16686_._.jaeger-query.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.20001_._.kiali.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.3000_._.grafana.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.31400_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.3306_._.mysql-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.3550_._.productcatalogservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.44134_._.tiller-deploy.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.kibana.logging.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.kubernetes-dashboard.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.kubernetes.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.metrics-server.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.443_._.zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.5000_._.emailservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.5000_._.orca.rca.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.5000_._.registry.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.50051_._.paymentservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.50051_._.shippingservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.5050_._.checkoutservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.53_._.coredns.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.6379_._.redis-cart.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.7000_._.currencyservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.7070_._.cartservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.8080_._.prometheus-kube-state-metrics.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.8080_._.prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.8080_._.recommendationservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.8080_._.web.default.svc.cluster.local"} 0
pilot_xds_eds_instances{cluster="outbound_.80_._.frontend-external.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.80_._.frontend.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.80_._.istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.80_._.istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.80_._.prometheus-grafana.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.80_._.tracing.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.80_._.zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.8529_._.arangodb.rca.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.853_._.istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.8765_._.falco.falco.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9080_._.details.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9080_._.productpage.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9080_._.ratings.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9080_._.reviews.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9090_._.prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9090_._.prometheus.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9093_._.prometheus-prometheus-oper-alertmanager.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9100_._.prometheus-prometheus-node-exporter.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9153_._.coredns.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9200_._.elasticsearch-client.logging.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9411_._.zipkin.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound_.9555_._.adservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|10050||zabbix-agent.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|10051||zabbix-proxy-mysql.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|10051||zabbix-proxy-sqlite3.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|10051||zabbix-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|10052||zabbix-java-gateway.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|14250||jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|14267||jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|14268||jaeger-collector.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|15010||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|15012||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|15014||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|15021||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|15443||istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|16686||jaeger-query.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|20001||kiali.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|3000||grafana.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|3306||mysql-server.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|3550||productcatalogservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|44134||tiller-deploy.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||kibana.logging.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||kubernetes-dashboard.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||kubernetes.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||metrics-server.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|443||zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|5000||emailservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|5000||orca.rca.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|5000||registry.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|50051||paymentservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|50051||shippingservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|5050||checkoutservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|53||coredns.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|6379||redis-cart.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|7000||currencyservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|7070||cartservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|8080||prometheus-kube-state-metrics.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|8080||prometheus-prometheus-oper-operator.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|8080||recommendationservice.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|8080||web.default.svc.cluster.local"} 0
pilot_xds_eds_instances{cluster="outbound|80||frontend-external.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|80||frontend.hipster-shop.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|80||istio-egressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|80||istio-ingressgateway.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|80||prometheus-grafana.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|80||tracing.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|80||zabbix-web.zabbix.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|8529||arangodb.rca.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|853||istiod.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|8765||falco.falco.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9080||details.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9080||productpage.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9080||ratings.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9080||reviews.default.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9090||prometheus-prometheus-oper-prometheus.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9090||prometheus.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9093||prometheus-prometheus-oper-alertmanager.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9100||prometheus-prometheus-node-exporter.monitoring.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9153||coredns.kube-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9200||elasticsearch-client.logging.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9411||zipkin.istio-system.svc.cluster.local"} 1
pilot_xds_eds_instances{cluster="outbound|9555||adservice.hipster-shop.svc.cluster.local"} 1
# HELP pilot_xds_push_time Total time in seconds Pilot takes to push lds, rds, cds and eds.
# TYPE pilot_xds_push_time histogram
pilot_xds_push_time_bucket{type="cds",le="0.01"} 437
pilot_xds_push_time_bucket{type="cds",le="0.1"} 448
pilot_xds_push_time_bucket{type="cds",le="1"} 448
pilot_xds_push_time_bucket{type="cds",le="3"} 448
pilot_xds_push_time_bucket{type="cds",le="5"} 448
pilot_xds_push_time_bucket{type="cds",le="10"} 448
pilot_xds_push_time_bucket{type="cds",le="20"} 448
pilot_xds_push_time_bucket{type="cds",le="30"} 448
pilot_xds_push_time_bucket{type="cds",le="+Inf"} 448
pilot_xds_push_time_sum{type="cds"} 0.9999464370000002
pilot_xds_push_time_count{type="cds"} 448
pilot_xds_push_time_bucket{type="eds",le="0.01"} 484
pilot_xds_push_time_bucket{type="eds",le="0.1"} 493
pilot_xds_push_time_bucket{type="eds",le="1"} 494
pilot_xds_push_time_bucket{type="eds",le="3"} 494
pilot_xds_push_time_bucket{type="eds",le="5"} 494
pilot_xds_push_time_bucket{type="eds",le="10"} 494
pilot_xds_push_time_bucket{type="eds",le="20"} 494
pilot_xds_push_time_bucket{type="eds",le="30"} 494
pilot_xds_push_time_bucket{type="eds",le="+Inf"} 494
pilot_xds_push_time_sum{type="eds"} 0.7184417909999988
pilot_xds_push_time_count{type="eds"} 494
pilot_xds_push_time_bucket{type="lds",le="0.01"} 432
pilot_xds_push_time_bucket{type="lds",le="0.1"} 447
pilot_xds_push_time_bucket{type="lds",le="1"} 450
pilot_xds_push_time_bucket{type="lds",le="3"} 450
pilot_xds_push_time_bucket{type="lds",le="5"} 450
pilot_xds_push_time_bucket{type="lds",le="10"} 450
pilot_xds_push_time_bucket{type="lds",le="20"} 450
pilot_xds_push_time_bucket{type="lds",le="30"} 450
pilot_xds_push_time_bucket{type="lds",le="+Inf"} 450
pilot_xds_push_time_sum{type="lds"} 1.7339818179999962
pilot_xds_push_time_count{type="lds"} 450
pilot_xds_push_time_bucket{type="rds",le="0.01"} 132
pilot_xds_push_time_bucket{type="rds",le="0.1"} 397
pilot_xds_push_time_bucket{type="rds",le="1"} 398
pilot_xds_push_time_bucket{type="rds",le="3"} 398
pilot_xds_push_time_bucket{type="rds",le="5"} 398
pilot_xds_push_time_bucket{type="rds",le="10"} 398
pilot_xds_push_time_bucket{type="rds",le="20"} 398
pilot_xds_push_time_bucket{type="rds",le="30"} 398
pilot_xds_push_time_bucket{type="rds",le="+Inf"} 398
pilot_xds_push_time_sum{type="rds"} 4.9822036239999985
pilot_xds_push_time_count{type="rds"} 398
# HELP pilot_xds_pushes Pilot build and send errors for lds, rds, cds and eds.
# TYPE pilot_xds_pushes counter
pilot_xds_pushes{type="cds"} 448
pilot_xds_pushes{type="eds"} 494
pilot_xds_pushes{type="lds"} 450
pilot_xds_pushes{type="rds"} 398
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 177.08
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 52
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 8.5065728e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.59835110759e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.95127808e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP sidecar_injection_requests_total Total number of Side car injection requests.
# TYPE sidecar_injection_requests_total counter
sidecar_injection_requests_total 6
# HELP sidecar_injection_success_total Total number of successful Side car injection requests.
# TYPE sidecar_injection_success_total counter
sidecar_injection_success_total 6

aleksandra-galara · 2020-09-03T07:17:30Z

Collected Envoy metrics:

# TYPE envoy_wasm_envoy_wasm_runtime_null_created counter
envoy_wasm_envoy_wasm_runtime_null_created{} 20
# TYPE envoy_wasm_vm_envoy_wasm_runtime_null_cloned counter
envoy_wasm_vm_envoy_wasm_runtime_null_cloned{} 0
# TYPE envoy_wasm_vm_envoy_wasm_runtime_null_created counter
envoy_wasm_vm_envoy_wasm_runtime_null_created{} 20
# TYPE envoy_metric_cache_count counter
envoy_metric_cache_count{cache="hit",wasm_filter="stats_filter"} 0
envoy_metric_cache_count{cache="miss",wasm_filter="stats_filter"} 5
# TYPE envoy_cluster_default_total_match_count counter
envoy_cluster_default_total_match_count{cluster_name="xds-grpc"} 104042
# TYPE envoy_cluster_lb_subsets_selected counter
envoy_cluster_lb_subsets_selected{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_total counter
envoy_cluster_upstream_cx_total{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_upstream_rq_per_try_timeout counter
envoy_cluster_upstream_rq_per_try_timeout{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_rx_reset counter
envoy_cluster_upstream_rq_rx_reset{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_tx_flush_timeout counter
envoy_cluster_http2_tx_flush_timeout{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_zone_no_capacity_left counter
envoy_cluster_lb_zone_no_capacity_left{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_zone_routing_cross_zone counter
envoy_cluster_lb_zone_routing_cross_zone{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_membership_change counter
envoy_cluster_membership_change{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_upstream_cx_overflow counter
envoy_cluster_upstream_cx_overflow{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_assignment_stale counter
envoy_cluster_assignment_stale{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_retry_success counter
envoy_cluster_upstream_rq_retry_success{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_ssl_session_reused counter
envoy_cluster_ssl_session_reused{cluster_name="xds-grpc"} 216
# TYPE envoy_cluster_upstream_internal_redirect_failed_total counter
envoy_cluster_upstream_internal_redirect_failed_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_zone_cluster_too_small counter
envoy_cluster_lb_zone_cluster_too_small{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_tx_reset counter
envoy_cluster_http2_tx_reset{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_close_notify counter
envoy_cluster_upstream_cx_close_notify{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_http2_inbound_empty_frames_flood counter
envoy_cluster_http2_inbound_empty_frames_flood{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_flow_control_backed_up_total counter
envoy_cluster_upstream_flow_control_backed_up_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_http2_total counter
envoy_cluster_upstream_cx_http2_total{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_internal_upstream_rq counter
envoy_cluster_internal_upstream_rq{response_code_class="5xx",cluster_name="xds-grpc"} 2
# TYPE envoy_cluster_lb_healthy_panic counter
envoy_cluster_lb_healthy_panic{cluster_name="xds-grpc"} 2
# TYPE envoy_cluster_ssl_fail_verify_san counter
envoy_cluster_ssl_fail_verify_san{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_flow_control_paused_reading_total counter
envoy_cluster_upstream_flow_control_paused_reading_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_pool_overflow counter
envoy_cluster_upstream_cx_pool_overflow{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_retry counter
envoy_cluster_upstream_rq_retry{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_update_failure counter
envoy_cluster_update_failure{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_http1_total counter
envoy_cluster_upstream_cx_http1_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_idle_timeout counter
envoy_cluster_upstream_cx_idle_timeout{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_client_ssl_socket_factory_downstream_context_secrets_not_ready counter
envoy_cluster_client_ssl_socket_factory_downstream_context_secrets_not_ready{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_destroy_remote_with_active_rq counter
envoy_cluster_upstream_cx_destroy_remote_with_active_rq{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_upstream_cx_rx_bytes_total counter
envoy_cluster_upstream_cx_rx_bytes_total{cluster_name="xds-grpc"} 111072767
# TYPE envoy_cluster_upstream_rq_pending_overflow counter
envoy_cluster_upstream_rq_pending_overflow{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_zone_routing_all_directly counter
envoy_cluster_lb_zone_routing_all_directly{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_internal_upstream_rq_completed counter
envoy_cluster_internal_upstream_rq_completed{cluster_name="xds-grpc"} 290
# TYPE envoy_cluster_ssl_connection_error counter
envoy_cluster_ssl_connection_error{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_retry_overflow counter
envoy_cluster_upstream_rq_retry_overflow{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_header_overflow counter
envoy_cluster_http2_header_overflow{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_total counter
envoy_cluster_upstream_rq_total{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_upstream_cx_protocol_error counter
envoy_cluster_upstream_cx_protocol_error{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_local_cluster_not_ok counter
envoy_cluster_lb_local_cluster_not_ok{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_max_requests counter
envoy_cluster_upstream_cx_max_requests{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_update_attempt counter
envoy_cluster_update_attempt{cluster_name="xds-grpc"} 104042
# TYPE envoy_cluster_ssl_handshake counter
envoy_cluster_ssl_handshake{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_http2_inbound_window_update_frames_flood counter
envoy_cluster_http2_inbound_window_update_frames_flood{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_subsets_fallback_panic counter
envoy_cluster_lb_subsets_fallback_panic{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_zone_routing_sampled counter
envoy_cluster_lb_zone_routing_sampled{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_original_dst_host_invalid counter
envoy_cluster_original_dst_host_invalid{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_tx_reset counter
envoy_cluster_upstream_rq_tx_reset{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq counter
envoy_cluster_upstream_rq{response_code_class="5xx",cluster_name="xds-grpc"} 2
# TYPE envoy_cluster_http2_outbound_control_flood counter
envoy_cluster_http2_outbound_control_flood{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_dropped_headers_with_underscores counter
envoy_cluster_http2_dropped_headers_with_underscores{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_outbound_flood counter
envoy_cluster_http2_outbound_flood{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_retry_limit_exceeded counter
envoy_cluster_upstream_rq_retry_limit_exceeded{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_flow_control_resumed_reading_total counter
envoy_cluster_upstream_flow_control_resumed_reading_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_trailers counter
envoy_cluster_http2_trailers{cluster_name="xds-grpc"} 0
envoy_cluster_internal_upstream_rq{response_code_class="2xx",cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_upstream_rq_503 counter
envoy_cluster_upstream_rq_503{cluster_name="xds-grpc"} 2
# TYPE envoy_cluster_ssl_no_certificate counter
envoy_cluster_ssl_no_certificate{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_rx_messaging_error counter
envoy_cluster_http2_rx_messaging_error{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_destroy_local counter
envoy_cluster_upstream_cx_destroy_local{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_rx_reset counter
envoy_cluster_http2_rx_reset{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_connect_fail counter
envoy_cluster_upstream_cx_connect_fail{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_cancelled counter
envoy_cluster_upstream_rq_cancelled{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_none_healthy counter
envoy_cluster_upstream_cx_none_healthy{cluster_name="xds-grpc"} 2
# TYPE envoy_cluster_upstream_rq_maintenance_mode counter
envoy_cluster_upstream_rq_maintenance_mode{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_pending_failure_eject counter
envoy_cluster_upstream_rq_pending_failure_eject{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_ssl_versions_TLSv1_2 counter
envoy_cluster_ssl_versions_TLSv1_2{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_ssl_curves_X25519 counter
envoy_cluster_ssl_curves_X25519{cluster_name="xds-grpc"} 288
envoy_cluster_upstream_rq{response_code_class="2xx",cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_ssl_fail_verify_no_cert counter
envoy_cluster_ssl_fail_verify_no_cert{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_timeout counter
envoy_cluster_upstream_rq_timeout{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_inbound_priority_frames_flood counter
envoy_cluster_http2_inbound_priority_frames_flood{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_requests_rejected_with_underscores_in_headers counter
envoy_cluster_http2_requests_rejected_with_underscores_in_headers{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_retry_or_shadow_abandoned counter
envoy_cluster_retry_or_shadow_abandoned{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_destroy_with_active_rq counter
envoy_cluster_upstream_cx_destroy_with_active_rq{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_upstream_internal_redirect_succeeded_total counter
envoy_cluster_upstream_internal_redirect_succeeded_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_connect_timeout counter
envoy_cluster_upstream_cx_connect_timeout{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_zone_number_differs counter
envoy_cluster_lb_zone_number_differs{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_completed counter
envoy_cluster_upstream_rq_completed{cluster_name="xds-grpc"} 290
# TYPE envoy_cluster_upstream_rq_200 counter
envoy_cluster_upstream_rq_200{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_lb_subsets_removed counter
envoy_cluster_lb_subsets_removed{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_internal_upstream_rq_503 counter
envoy_cluster_internal_upstream_rq_503{cluster_name="xds-grpc"} 2
# TYPE envoy_cluster_update_success counter
envoy_cluster_update_success{cluster_name="xds-grpc"} 104042
# TYPE envoy_cluster_upstream_cx_destroy_local_with_active_rq counter
envoy_cluster_upstream_cx_destroy_local_with_active_rq{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_ssl_ciphers_ECDHE_RSA_AES128_GCM_SHA256 counter
envoy_cluster_ssl_ciphers_ECDHE_RSA_AES128_GCM_SHA256{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_upstream_rq_pending_total counter
envoy_cluster_upstream_rq_pending_total{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_ssl_fail_verify_cert_hash counter
envoy_cluster_ssl_fail_verify_cert_hash{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_connect_attempts_exceeded counter
envoy_cluster_upstream_cx_connect_attempts_exceeded{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_subsets_fallback counter
envoy_cluster_lb_subsets_fallback{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_recalculate_zone_structures counter
envoy_cluster_lb_recalculate_zone_structures{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_flow_control_drained_total counter
envoy_cluster_upstream_flow_control_drained_total{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_bind_errors counter
envoy_cluster_bind_errors{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_subsets_created counter
envoy_cluster_lb_subsets_created{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_update_empty counter
envoy_cluster_update_empty{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_update_no_rebuild counter
envoy_cluster_update_no_rebuild{cluster_name="xds-grpc"} 104041
# TYPE envoy_cluster_http2_headers_cb_no_stream counter
envoy_cluster_http2_headers_cb_no_stream{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_client_ssl_socket_factory_upstream_context_secrets_not_ready counter
envoy_cluster_client_ssl_socket_factory_upstream_context_secrets_not_ready{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_client_ssl_socket_factory_ssl_context_update_by_sds counter
envoy_cluster_client_ssl_socket_factory_ssl_context_update_by_sds{cluster_name="xds-grpc"} 13
# TYPE envoy_cluster_ssl_sigalgs_rsa_pss_rsae_sha256 counter
envoy_cluster_ssl_sigalgs_rsa_pss_rsae_sha256{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_http2_too_many_header_frames counter
envoy_cluster_http2_too_many_header_frames{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_internal_upstream_rq_200 counter
envoy_cluster_internal_upstream_rq_200{cluster_name="xds-grpc"} 288
# TYPE envoy_cluster_upstream_cx_destroy_remote counter
envoy_cluster_upstream_cx_destroy_remote{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_upstream_cx_tx_bytes_total counter
envoy_cluster_upstream_cx_tx_bytes_total{cluster_name="xds-grpc"} 29465246
# TYPE envoy_cluster_ssl_fail_verify_error counter
envoy_cluster_ssl_fail_verify_error{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_destroy counter
envoy_cluster_upstream_cx_destroy{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_assignment_timeout_received counter
envoy_cluster_assignment_timeout_received{cluster_name="xds-grpc"} 0
# TYPE envoy_listener_manager_listener_added counter
envoy_listener_manager_listener_added{} 75
# TYPE envoy_listener_manager_listener_create_failure counter
envoy_listener_manager_listener_create_failure{} 0
# TYPE envoy_listener_manager_listener_modified counter
envoy_listener_manager_listener_modified{} 0
# TYPE envoy_listener_manager_listener_removed counter
envoy_listener_manager_listener_removed{} 1
# TYPE envoy_listener_manager_listener_stopped counter
envoy_listener_manager_listener_stopped{} 0
# TYPE envoy_listener_manager_listener_in_place_updated counter
envoy_listener_manager_listener_in_place_updated{} 0
# TYPE envoy_listener_manager_listener_create_success counter
envoy_listener_manager_listener_create_success{} 150
# TYPE istio_requests_total counter
istio_requests_total{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 8
istio_requests_total{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 7
# TYPE envoy_cluster_manager_cluster_updated_via_merge counter
envoy_cluster_manager_cluster_updated_via_merge{} 0
# TYPE envoy_server_dynamic_unknown_fields counter
envoy_server_dynamic_unknown_fields{} 0
# TYPE envoy_server_worker_0_watchdog_mega_miss counter
envoy_server_worker_0_watchdog_mega_miss{} 0
# TYPE envoy_cluster_manager_cluster_updated counter
envoy_cluster_manager_cluster_updated{} 1
# TYPE envoy_cluster_manager_update_merge_cancelled counter
envoy_cluster_manager_update_merge_cancelled{} 0
# TYPE envoy_server_worker_0_watchdog_miss counter
envoy_server_worker_0_watchdog_miss{} 0
# TYPE envoy_cluster_manager_update_out_of_merge_window counter
envoy_cluster_manager_update_out_of_merge_window{} 0
# TYPE envoy_server_main_thread_watchdog_miss counter
envoy_server_main_thread_watchdog_miss{} 0
# TYPE envoy_server_static_unknown_fields counter
envoy_server_static_unknown_fields{} 0
# TYPE envoy_cluster_manager_cluster_modified counter
envoy_cluster_manager_cluster_modified{} 6
# TYPE envoy_server_worker_1_watchdog_mega_miss counter
envoy_server_worker_1_watchdog_mega_miss{} 0
# TYPE envoy_server_watchdog_miss counter
envoy_server_watchdog_miss{} 0
# TYPE envoy_server_worker_1_watchdog_miss counter
envoy_server_worker_1_watchdog_miss{} 0
# TYPE envoy_server_watchdog_mega_miss counter
envoy_server_watchdog_mega_miss{} 0
# TYPE envoy_server_debug_assertion_failures counter
envoy_server_debug_assertion_failures{} 0
# TYPE envoy_server_main_thread_watchdog_mega_miss counter
envoy_server_main_thread_watchdog_mega_miss{} 0
# TYPE envoy_cluster_manager_cluster_added counter
envoy_cluster_manager_cluster_added{} 105
# TYPE envoy_cluster_manager_cluster_removed counter
envoy_cluster_manager_cluster_removed{} 1
istio_requests_total{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 1
# TYPE envoy_listener_manager_lds_update_failure counter
envoy_listener_manager_lds_update_failure{} 287
# TYPE envoy_listener_manager_lds_update_success counter
envoy_listener_manager_lds_update_success{} 290
# TYPE envoy_listener_manager_lds_update_rejected counter
envoy_listener_manager_lds_update_rejected{} 0
# TYPE envoy_listener_manager_lds_update_attempt counter
envoy_listener_manager_lds_update_attempt{} 578
# TYPE envoy_listener_manager_lds_init_fetch_timeout counter
envoy_listener_manager_lds_init_fetch_timeout{} 0
# TYPE envoy_cluster_manager_cds_update_rejected counter
envoy_cluster_manager_cds_update_rejected{} 0
# TYPE envoy_cluster_manager_cds_init_fetch_timeout counter
envoy_cluster_manager_cds_init_fetch_timeout{} 0
# TYPE envoy_cluster_manager_cds_update_success counter
envoy_cluster_manager_cds_update_success{} 291
# TYPE envoy_cluster_manager_cds_update_attempt counter
envoy_cluster_manager_cds_update_attempt{} 579
# TYPE envoy_cluster_manager_cds_update_failure counter
envoy_cluster_manager_cds_update_failure{} 287
# TYPE envoy_wasm_envoy_wasm_runtime_null_active gauge
envoy_wasm_envoy_wasm_runtime_null_active{} 20
# TYPE envoy_wasm_vm_envoy_wasm_runtime_null_active gauge
envoy_wasm_vm_envoy_wasm_runtime_null_active{} 20
# TYPE istio_build gauge
istio_build{component="proxy",tag="1.6.8"} 1
# TYPE envoy_cluster_upstream_rq_pending_active gauge
envoy_cluster_upstream_rq_pending_active{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_active gauge
envoy_cluster_upstream_cx_active{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_circuit_breakers_high_rq_open gauge
envoy_cluster_circuit_breakers_high_rq_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_circuit_breakers_default_rq_pending_open gauge
envoy_cluster_circuit_breakers_default_rq_pending_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_version gauge
envoy_cluster_version{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_max_host_weight gauge
envoy_cluster_max_host_weight{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_circuit_breakers_default_cx_pool_open gauge
envoy_cluster_circuit_breakers_default_cx_pool_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_rq_active gauge
envoy_cluster_upstream_rq_active{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_upstream_cx_tx_bytes_buffered gauge
envoy_cluster_upstream_cx_tx_bytes_buffered{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_circuit_breakers_default_cx_open gauge
envoy_cluster_circuit_breakers_default_cx_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_membership_degraded gauge
envoy_cluster_membership_degraded{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_circuit_breakers_default_rq_open gauge
envoy_cluster_circuit_breakers_default_rq_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_circuit_breakers_high_rq_pending_open gauge
envoy_cluster_circuit_breakers_high_rq_pending_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_membership_total gauge
envoy_cluster_membership_total{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_http2_streams_active gauge
envoy_cluster_http2_streams_active{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_circuit_breakers_high_rq_retry_open gauge
envoy_cluster_circuit_breakers_high_rq_retry_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_http2_pending_send_bytes gauge
envoy_cluster_http2_pending_send_bytes{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_lb_subsets_active gauge
envoy_cluster_lb_subsets_active{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_membership_excluded gauge
envoy_cluster_membership_excluded{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_upstream_cx_rx_bytes_buffered gauge
envoy_cluster_upstream_cx_rx_bytes_buffered{cluster_name="xds-grpc"} 17
# TYPE envoy_cluster_circuit_breakers_default_rq_retry_open gauge
envoy_cluster_circuit_breakers_default_rq_retry_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_circuit_breakers_high_cx_open gauge
envoy_cluster_circuit_breakers_high_cx_open{cluster_name="xds-grpc"} 0
# TYPE envoy_cluster_membership_healthy gauge
envoy_cluster_membership_healthy{cluster_name="xds-grpc"} 1
# TYPE envoy_cluster_circuit_breakers_high_cx_pool_open gauge
envoy_cluster_circuit_breakers_high_cx_pool_open{cluster_name="xds-grpc"} 0
# TYPE envoy_listener_manager_workers_started gauge
envoy_listener_manager_workers_started{} 1
# TYPE envoy_listener_manager_total_listeners_warming gauge
envoy_listener_manager_total_listeners_warming{} 0
# TYPE envoy_listener_manager_total_listeners_draining gauge
envoy_listener_manager_total_listeners_draining{} 0
# TYPE envoy_listener_manager_total_filter_chains_draining gauge
envoy_listener_manager_total_filter_chains_draining{} 0
# TYPE envoy_listener_manager_total_listeners_active gauge
envoy_listener_manager_total_listeners_active{} 74
# TYPE envoy_server_memory_physical_size gauge
envoy_server_memory_physical_size{} 28835840
# TYPE envoy_server_memory_allocated gauge
envoy_server_memory_allocated{} 20709424
# TYPE envoy_server_memory_heap_size gauge
envoy_server_memory_heap_size{} 26214400
# TYPE envoy_server_stats_recent_lookups gauge
envoy_server_stats_recent_lookups{} 0
# TYPE envoy_server_version gauge
envoy_server_version{} 13413915
# TYPE envoy_server_uptime gauge
envoy_server_uptime{} 519491
# TYPE envoy_server_live gauge
envoy_server_live{} 1
# TYPE envoy_server_days_until_first_cert_expiring gauge
envoy_server_days_until_first_cert_expiring{} 0
# TYPE envoy_cluster_manager_active_clusters gauge
envoy_cluster_manager_active_clusters{} 104
# TYPE envoy_cluster_manager_warming_clusters gauge
envoy_cluster_manager_warming_clusters{} 0
# TYPE envoy_server_total_connections gauge
envoy_server_total_connections{} 1
# TYPE envoy_server_parent_connections gauge
envoy_server_parent_connections{} 0
# TYPE envoy_server_concurrency gauge
envoy_server_concurrency{} 2
# TYPE envoy_server_hot_restart_generation gauge
envoy_server_hot_restart_generation{} 1
# TYPE envoy_server_hot_restart_epoch gauge
envoy_server_hot_restart_epoch{} 0
# TYPE envoy_server_state gauge
envoy_server_state{} 0
# TYPE envoy_listener_manager_lds_update_time gauge
envoy_listener_manager_lds_update_time{} 1598869542541
# TYPE envoy_listener_manager_lds_version gauge
envoy_listener_manager_lds_version{} 8672229125421287002
# TYPE envoy_cluster_manager_cds_version gauge
envoy_cluster_manager_cds_version{} 8672229125421287002
# TYPE envoy_cluster_manager_cds_update_time gauge
envoy_cluster_manager_cds_update_time{} 1598869542509
# TYPE envoy_cluster_upstream_cx_length_ms histogram
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="0.5"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="1"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="5"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="10"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="25"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="50"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="100"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="250"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="500"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="1000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="2500"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="5000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="10000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="30000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="60000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="300000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="600000"} 0
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="1800000"} 150
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="3600000"} 287
envoy_cluster_upstream_cx_length_ms_bucket{cluster_name="xds-grpc",le="+Inf"} 287
envoy_cluster_upstream_cx_length_ms_sum{cluster_name="xds-grpc"} 517550000
envoy_cluster_upstream_cx_length_ms_count{cluster_name="xds-grpc"} 287
# TYPE envoy_cluster_upstream_cx_connect_ms histogram
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="0.5"} 167
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1"} 167
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="5"} 279
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="10"} 287
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="25"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="50"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="100"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="250"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="500"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="2500"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="5000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="10000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="30000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="60000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="300000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="600000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1800000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="3600000"} 288
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="+Inf"} 288
envoy_cluster_upstream_cx_connect_ms_sum{cluster_name="xds-grpc"} 331.5
envoy_cluster_upstream_cx_connect_ms_count{cluster_name="xds-grpc"} 288
# TYPE istio_request_duration_milliseconds histogram
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="0.5"} 3
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1"} 3
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="5"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="10"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="25"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="50"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="100"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="250"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="500"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="2500"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="5000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="10000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="30000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="60000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="300000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="600000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1800000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="3600000"} 8
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="+Inf"} 8
istio_request_duration_milliseconds_sum{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 9.25
istio_request_duration_milliseconds_count{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 8
# TYPE istio_response_bytes histogram
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="0.5"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="5"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="10"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="25"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="50"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="100"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="250"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="500"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1000"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="2500"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="5000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="10000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="30000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="60000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="300000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="600000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1800000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="3600000"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="+Inf"} 8
istio_response_bytes_sum{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 8400
istio_response_bytes_count{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 8
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="0.5"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="5"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="10"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="25"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="50"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="100"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="250"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="500"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1000"} 0
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="2500"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="5000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="10000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="30000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="60000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="300000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="600000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1800000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="3600000"} 7
istio_response_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="+Inf"} 7
istio_response_bytes_sum{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 7350
istio_response_bytes_count{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="0.5"} 3
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1"} 3
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="5"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="10"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="25"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="50"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="100"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="250"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="500"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="2500"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="5000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="10000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="30000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="60000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="300000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="600000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1800000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="3600000"} 7
istio_request_duration_milliseconds_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="+Inf"} 7
istio_request_duration_milliseconds_sum{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 4.200000000000000177635683940025
istio_request_duration_milliseconds_count{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 7
# TYPE istio_request_bytes histogram
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="0.5"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="5"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="10"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="25"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="50"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="100"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="250"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="500"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="2500"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="5000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="10000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="30000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="60000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="300000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="600000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="1800000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="3600000"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1",le="+Inf"} 8
istio_request_bytes_sum{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 5060
istio_request_bytes_count{response_code="200",reporter="destination",source_workload="reviews-v3",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v3",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v3",destination_canonical_revision="v1"} 8
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="0.5"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="5"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="10"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="25"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="50"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="100"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="250"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="500"} 0
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="2500"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="5000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="10000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="30000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="60000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="300000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="600000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="1800000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="3600000"} 7
istio_request_bytes_bucket{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1",le="+Inf"} 7
istio_request_bytes_sum{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 4485
istio_request_bytes_count{response_code="200",reporter="destination",source_workload="reviews-v2",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-reviews",source_app="reviews",source_version="v2",destination_workload="ratings-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",destination_app="ratings",destination_version="v1",destination_service="ratings.default.svc.cluster.local",destination_service_name="ratings",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="reviews",destination_canonical_service="ratings",source_canonical_revision="v2",destination_canonical_revision="v1"} 7
# TYPE envoy_server_initialization_time_ms histogram
envoy_server_initialization_time_ms_bucket{le="0.5"} 0
envoy_server_initialization_time_ms_bucket{le="1"} 0
envoy_server_initialization_time_ms_bucket{le="5"} 0
envoy_server_initialization_time_ms_bucket{le="10"} 0
envoy_server_initialization_time_ms_bucket{le="25"} 0
envoy_server_initialization_time_ms_bucket{le="50"} 0
envoy_server_initialization_time_ms_bucket{le="100"} 0
envoy_server_initialization_time_ms_bucket{le="250"} 0
envoy_server_initialization_time_ms_bucket{le="500"} 0
envoy_server_initialization_time_ms_bucket{le="1000"} 0
envoy_server_initialization_time_ms_bucket{le="2500"} 1
envoy_server_initialization_time_ms_bucket{le="5000"} 1
envoy_server_initialization_time_ms_bucket{le="10000"} 1
envoy_server_initialization_time_ms_bucket{le="30000"} 1
envoy_server_initialization_time_ms_bucket{le="60000"} 1
envoy_server_initialization_time_ms_bucket{le="300000"} 1
envoy_server_initialization_time_ms_bucket{le="600000"} 1
envoy_server_initialization_time_ms_bucket{le="1800000"} 1
envoy_server_initialization_time_ms_bucket{le="3600000"} 1
envoy_server_initialization_time_ms_bucket{le="+Inf"} 1
envoy_server_initialization_time_ms_sum{} 1350
envoy_server_initialization_time_ms_count{} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="0.5"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="5"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="10"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="25"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="50"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="100"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="250"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="500"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1000"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="2500"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="5000"} 0
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="10000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="30000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="60000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="300000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="600000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1800000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="3600000"} 1
istio_response_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="+Inf"} 1
istio_response_bytes_sum{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 5350
istio_response_bytes_count{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="0.5"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="5"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="10"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="25"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="50"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="100"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="250"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="500"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1000"} 0
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="2500"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="5000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="10000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="30000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="60000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="300000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="600000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1800000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="3600000"} 1
istio_request_bytes_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="+Inf"} 1
istio_request_bytes_sum{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 1150
istio_request_bytes_count{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="0.5"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="5"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="10"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="25"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="50"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="100"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="250"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="500"} 0
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="2500"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="5000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="10000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="30000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="60000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="300000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="600000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="1800000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="3600000"} 1
istio_request_duration_milliseconds_bucket{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1",le="+Inf"} 1
istio_request_duration_milliseconds_sum{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 735
istio_request_duration_milliseconds_count{response_code="200",reporter="source",source_workload="ratings-v1",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/bookinfo-ratings",source_app="ratings",source_version="v1",destination_workload="productpage-v1",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",destination_app="productpage",destination_version="v1",destination_service="productpage.default.svc.cluster.local",destination_service_name="productpage",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="ratings",destination_canonical_service="productpage",source_canonical_revision="v1",destination_canonical_revision="v1"} 1

aleksandra-galara · 2020-09-03T07:17:43Z

Alerts draft for Citadel, Galley and Pilot (will be updated)

**CITADEL:**
- alert: IstioCitadelRootCertError
 annotations:
      summary: "Citadel root certificate internal error occured"
      description: "Citadel root certificate internal error occured on {{ $labels.instance }} instance"
  expr: >
	citadel_server_root_cert_expiry_timestamp <0
  
- alert: IstioCitadelAuthenticationFailure
 annotations:
      summary: "Citadel server authentication failed"
      description: "Citadel server authentication failed in last 5 minutes on {{ $labels.instance }} instance"
  expr: >
	(absent(citadel_server_authentication_failure_count offset 5m) == 1 and citadel_server_authentication_failure_count > 0) or (citadel_server_authentication_failure_count - citadel_server_authentication_failure_count 5m > 0)
  
- alert: IstioCitadelCsrParsingError
  annotations:
      summary: "Citadel CSR parsing error"
      description: "Citadel CSR parsing error occured in last 5 minutes on {{ $labels.instance }} instance"
  expr: >
	(absent(citadel_server_csr_parsing_err_count offset 5m) == 1 and citadel_server_csr_parsing_err_count > 0) or (citadel_server_csr_parsing_err_count - citadel_server_csr_parsing_err_count offset 5m > 0)
 
- alert: IstioCitadelCsrSignError
 annotations:
      summary: "Citadel CSR signing error"
      description: "Citadel CSR signing error occured in last 5 minutes on {{ $labels.instance }} instance"
  expr: >
	(absent(citadel_server_csr_sign_err_count offset 5m) == 1 and citadel_server_csr_sign_err_count > 0) or (citadel_server_csr_sign_err_count - citadel_server_csr_sign_err_count offset 5m > 0)

- alert: IstioCitadelCertIssuanceFailure
 annotations:
      summary: "Citadel certificate issuance failed"
      description: "Citadel certificate issuance failed in last 5 minutes"
expr: >
	(citadel_server_csr_count - citadel_server_success_cert_issuance_count) > (citadel_server_csr_count offset 5m - citadel_server_success_cert_issuance_count offset 5m)
 

**GALLEY**:
- alert: IstioGalleyConfigUpdateError
 annotations:
      summary: "Galley configuration upadate error occured"
      description: "Galley configuration upadate error occured in last 5 minutes."
  expr: >
	(absent(galley_validation_config_update_error offset 5m) == 1 and galley_validation_config_update_error > 0) or (galley_validation_config_update_error - galley_validation_config_update_error offset 5m > 0)
 
- alert: IstioGalleyConfigLoadError
 annotations:
      summary: "Galley configuration loading error occured"
      description: "Galley configuration loading error occured in last 5 minutes."
  expr: >
	(absent(galley_validation_config_load_error offset 5m) == 1 and galley_validation_config_load_error > 0) or (galley_validation_config_load_error - galley_validation_config_load_error offset 5m > 0)
 
- alert: IstioGalleyCertKeyUpdateError
 annotations:
      summary: "Galley certificate key upadate error occured"
      description: "Galley certificate key update error occured in last 5 minutes"
  expr: >
	(absent(galley_validation_cert_key_update_errors offset 5m) == 1 and galley_validation_cert_key_update_errors > 0) or (galley_validation_cert_key_update_errors - galley_validation_cert_key_update_errors offset 5m > 0)
 
- alert: IstioGalleyValidationFailed
 annotations:
      summary: "Galley validation failed"
      description: "Galley validation failed in last 5 minutes"
  expr: >
	(absent(galley_validation_failed offset 5m) == 1 and galley_validation_failed > 0) or (galley_validation_failed - galley_validation_failed offset 5m > 0)


**PILOT**:

- alert: IstioPilotPodNotInEndpointTable
 annotations:
      summary: "Pilot pods not found in the endpoint table"
      description: "Pods not found in the endpoint table, possibly invalid"
  expr: >
	pilot_no_ip > 0
  
- alert: IstioPilotEndpointNotReady
 annotations: 
      summary: "Pilot endpoint found in unready state"
      description: "Pilot endpoint found in unready state for 3 minutes"
  expr: >
	pilot_endpoint_not_ready > 0
 for: 3m
 
- alert: IstioPilotNoCorrespondingPodsForEndpoint
 annotations:
      summary: "Not corresponding pods for endpoint"
      description: "Some of pilot endpoints do not currently have any corresponding pods"
  expr: >
	pilot_k8s_endpoints_pending_pod > 0
 
- alert: IstioPilotXDSInternalError
 annotations:
      summary: "Pilot XDS internal error"
      descripription: "Pilot XDS internal error occured in last 5m"
  expr: >
	pilot_total_xds_internal_errors - pilot_total_xds_internal_errors offset 5m > 0
  for:
- alert: IstioPilotCRDConvertError
 annotations:
      summary: "Pilot k8s CRD convertion errorr"
      description: "Pilot k8s CRD convertion error occured in last 5 minutes"
  expr: >
	pilot_total_k8s_object_errors - pilot_total_k8s_object_errors offset 5m > 0
  for:
- alert: IstioPilotNoXDSInstances
 annotations: 
      summary: "No XDS instances for cluster"
      description: "Error occured.  {{ $labels.cluster }} cluster has no XDS instances."
  expr: >
	pilot_xds_eds_instances == 0
  for 3m

- alert: IstioPilotNoXDSEndpoints
 annotations:
      summary: "No network XDS endpoints for cluster"
      description: "Error occured. {{ $labels.cluster }} cluster has no XDS endpoints."
  expr: >
	pilot_xds_eds_all_locality_endpoints == 0
  
- alert: IstioPilotInvalidOutboundListener
 annotations:
      summary: "Invalid outbound listener"
      descripton: " Pilot {{ $labels.instance }} instance  has some invalid outbound listeners"
  expr: >
	pilot_invalid_out_listeners > 0

- alert: IstioSidecarInjectionFailure:
 annotations:
      summary: "Istio sidecar injection failure"
      description: "Istio sidecar injection failed in last 5 minutes"
  expr: >
	sidecar_injection_requests_total - sidecar_injection_success_total > sidecar_injection_requests_total offset 5m - sidecar_injection_success_total offset 5m


**ISTIO**:

- alert: IstioHighRequestDuration
annotations:
      summary: "Istio request duration is too high"
      description: "Istio request duration between {{ $labels.source_app }} and {{ $labels.destination_app }} is higher than usual in last 5 minutes"
  expr: >
 sum by (destination_app, source_app, instance) ( istio_request_duration_milliseconds_sum - istio_request_duration_milliseconds_sum offset 5m )/ sum by (destination_app, source_app, instance) ( istio_request_duration_milliseconds_count - istio_request_duration_milliseconds_count offset 5m)  > 2 * sum by (destination_app, source_app, instance) (istio_request_duration_milliseconds_sum offset 5m)/sum by (destination_app, source_app, instance) (istio_request_duration_milliseconds_count offset 5m)
for: 5m

- alert: IstioLowRequestDuration
annotations:  
      summary: "Istio request duration is too low"
      description: "Istio request duration between {{ $labels.source_app }} and {{ $labels.destination_app }} is lower than usual for last 5 minutes"
  expr: >
  sum by (destination_app, source_app, instance) ( istio_request_duration_milliseconds_sum - istio_request_duration_milliseconds_sum offset 5m )/ sum by (destination_app, source_app, instance) ( istio_request_duration_milliseconds_count - istio_request_duration_milliseconds_count offset 5m)  <  0.3 * sum by (destination_app, source_app, instance) (istio_request_duration_milliseconds_sum offset 5m)/sum by (destination_app, source_app, instance) (istio_request_duration_milliseconds_count offset 5m)
for: 5m 

- alert: IstioAbnormalResponseSize
annotations:
      summary:  "Istio response size is too low"
      descrition:  "Istio responses size between {{ $labels.source_app }} and {{ $labels.destination_app }} is lower than usual for last 5 minutes"
  expr: >
sum by (destination_app, source_app, instance) (istio_response_bytes_sum offset 5m)/ sum by (destination_app, source_app, instance) (istio_response_bytes_count offset 5m) > 1.5 * sum by (destination_app, source_app, instance)(istio_response_bytes_sum - istio_response_bytes_sum offset 5m)/sum by (destination_app, source_app, instance)(istio_response_bytes_count - istio_response_bytes_count offset 5m)

- alert: IstioRequestTotalHTTP5xxRateHigh
annotations:
      summary: 'Istio Percentage of HTTP 5xx responses is too high'
      description: "The amount of 5xx responses between {{ $labels.source_app }} and {{ $labels.destination_app }}  is higher than 5 percentage. It is too high."
  expr: >
sum by (destination_app, source_app, instance) (rate(istio_requests_total{response_code=~"5.*"}[5m]))/sum by (destination_app, source_app, instance) (rate(istio_requests_total[5m])) > 0.05

- alert: IstioRequestTotalHTTP200RateLow
annotations:
      summary: 'Istio Percentage of HTTP 2xx responses is too low'
      description: "The amount of 2xx responses between {{ $labels.source_app }} and {{ $labels.destination_app }}  is lower than 95 percentage. It is too low."
  expr: >
sum by (destination_app, source_app, instance) (rate(istio_requests_total{response_code=~"2.*"}[5m]))/sum by (destination_app, source_app, instance) (rate(istio_requests_total[5m])) < 0.95

Alerts draft for Envoy (will be updated)

- alert: IstioEnvoyInternalUpstreamReq503TooHigh
 annotations:
      summary: ' Envoy Percentage of HTTP 503 internal upstream responses is too high'
      description: "The amount of 503 internal upstream responses is higher than 5%. It is too high"
  expr: >
	rate(envoy_cluster_internal_upstream_rq_503[5m])/rate(envoy_cluster_internal_upstream_rq_completed[5m]) > 0.05

- alert: IstioEnvoyInternalUpstreamReq200TooLow
annotations:
      summary: 'Envoy Percentage of HTTP 200 internal upstream responses is too low'
      description: "The amount of 200 internal upstream responses is lower than 95%. It is too low"
  expr: >
	rate(envoy_cluster_internal_upstream_rq_200[5m])/rate(envoy_cluster_internal_upstream_rq_completed[5m]) < 0.95

- alert: IstioEnvoyUpstreamReq503TooHigh
annotations:
      summary: ' Envoy Percentage of HTTP 503 upstream responses is too high'
      description: "The amount of 503 upstream responses is higher than 5%. It is too high"
  expr: >
	rate(envoy_cluster_upstream_rq_503[5m])/rate(envoy_cluster_upstream_rq_completed[5m]) > 0.05

- alert: IstioEnvoyUpstreamReq200TooLow
annotations:
      summary: 'Envoy Percentage of HTTP 200 upstream responses is too low'
      description: "The amount of 200 upstream responses is lower than 95%. It is too low"
  expr: >
	rate(envoy_cluster_upstream_rq_200[5m])/rate(envoy_cluster_upstream_rq_completed[5m]) < 0.95

- alert: IstioEnvoyClusterBindErrors
annotations:
      summary: "Envoy cluster binding errors"
       description: "Error in binding cluster with {{ $labels.pod_name }}  pod in {{ $labels.namespace }} namespace."
  expr: >
	envoy_cluster_bind_errors > 0

- alert: IstioEnvoyClusterDstHostInvalid
annotations:
      summary: "Envoy cluster destination host invalid"
      description: "Envoy cluster destination host {{ $labels.pod_name }} in {{ $labels.namespace }} namespace invalid for 3 minutes"
  expr: >
	envoy_cluster_original_dst_host_invalid > 0
For: 3m

- alert: IstioEnvoyDownstreamSSLSecretsNotReady
annotations:
      summary: "Envoy downstream SSL secrets found in not ready state"
      description: "Envoy downstream SSL secrets found for {{ $labels.pod_name }} in {{ $labels.namespace }} namespace in not ready state for 3 minutes"
  expr: >
	envoy_cluster_client_ssl_socket_factory_downstream_context_secrets_not_ready > 0
For: 3m

- alert: IstioEnvoyUpstreamSSLSecretsNotReady
annotations:
      summary: "Envoy upstream SSL secrets found in not ready state"
      description: "Envoy upstream SSL secrets found for {{ $labels.pod_name }} in {{ $labels.namespace }} namespace in not ready state for 3 minutes"
  expr: >
	envoy_cluster_client_ssl_socket_factory_upstream_context_secrets_not_ready > 0
For: 3m

eduardobaitello · 2022-11-07T16:18:32Z

I'm running Istio 1.14.1 on a test environment, and can't find some of these metrics. It looks like IstioD is not exporting them:

citadel_server_authentication_failure_count
citadel_server_csr_parsing_err_count
citadel_server_csr_sign_err_count
galley_validation_config_load_error
galley_validation_cert_key_update_errors
pilot_total_xds_internal_errors
pilot_total_k8s_object_errors
pilot_xds_eds_instances
pilot_xds_eds_all_locality_endpoints
pilot_xds_push_errors

Some of these metrics are not even documented here.

For these _errors metrics, should they be absent in case of no errors, or exported with a 0 value?

bzurkowski · 2022-11-07T16:30:59Z

@eduardobaitello The above alert rules for Istio were created almost 2 years ago - these are lightyears in the case of Istio. The set of supported Istio metrics must have been changed or RENAMED over time.

bzurkowski added the enhancement New feature or request label Apr 26, 2020

bzurkowski assigned aleksandra-galara Apr 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prepare Prometheus alerting rules for Istio #65

Prepare Prometheus alerting rules for Istio #65

bzurkowski commented Apr 26, 2020 •

edited

Loading

bzurkowski commented Apr 26, 2020

bzurkowski commented Apr 26, 2020

bzurkowski commented Apr 26, 2020

bzurkowski commented Aug 7, 2020 •

edited

Loading

aleksandra-galara commented Aug 26, 2020

aleksandra-galara commented Sep 3, 2020

aleksandra-galara commented Sep 3, 2020

aleksandra-galara commented Sep 3, 2020 •

edited

Loading

eduardobaitello commented Nov 7, 2022

bzurkowski commented Nov 7, 2022

Prepare Prometheus alerting rules for Istio #65

Prepare Prometheus alerting rules for Istio #65

Comments

bzurkowski commented Apr 26, 2020 • edited Loading

bzurkowski commented Apr 26, 2020

bzurkowski commented Apr 26, 2020

bzurkowski commented Apr 26, 2020

bzurkowski commented Aug 7, 2020 • edited Loading

aleksandra-galara commented Aug 26, 2020

aleksandra-galara commented Sep 3, 2020

aleksandra-galara commented Sep 3, 2020

aleksandra-galara commented Sep 3, 2020 • edited Loading

eduardobaitello commented Nov 7, 2022

bzurkowski commented Nov 7, 2022

bzurkowski commented Apr 26, 2020 •

edited

Loading

bzurkowski commented Aug 7, 2020 •

edited

Loading

aleksandra-galara commented Sep 3, 2020 •

edited

Loading