Final monitoring doc revision (#433)

* Correct helm chart option

* Revise monitoring docs

* update howto readme

* update concepts

* fix cr

* wip

* Fix dashboard for minikube

* Fix steps for prometheus howto

* Update ordering

* fix ct

* wip

* Update README.md

Author: youngbupark

concepts/observability/README.md

@@ -1,6 +1,6 @@
 # Observability
 
-Observability is a term from control theory. Observability means you can answer questions about what’s happening on the inside of a system by observing the outside of the system, without having to ship new code to answer new questions. Observability is critical in production environments and services to debug, operate and monitor Dapr system services, components and user applications. 
+Observability is a term from control theory. Observability means you can answer questions about what's happening on the inside of a system by observing the outside of the system, without having to ship new code to answer new questions. Observability is critical in production environments and services to debug, operate and monitor Dapr system services, components and user applications.
 
 The observability capabilities enable users to monitor the Dapr system services, their interaction with user applications and understand how these monitored services behave. The observability capabilities are divided into three main areas;
 
@@ -17,8 +17,9 @@ The table below shows the current status of each of the observabilty capabilites
 |Tracing  | Yes     | N/A      | N/A      | *Planned* | N/A    |
 |Logs     | Yes     | Yes      | Yes      | Yes       | Yes    |
 
-## Supported monitoring tools
-The observability tools listed below are ones that have been tested to work with Dapr
+## Monitoring tools
+
+The observability tools listed below are ones that have been tested to work with Dapr. 
 
 ### Metrics
 

concepts/observability/logs.md

@@ -44,10 +44,10 @@ The following steps describe how to configure JSON formatted logs for Kubernetes
 
 ### Install Dapr to your cluster using the Helm chart
 
-You can enable JSON formatted logs for Dapr system services by adding `--set global.LogASJSON=true` option to Helm command.
+You can enable JSON formatted logs for Dapr system services by adding `--set global.logAsJson=true` option to Helm command.
 
 ```bash
-helm install dapr dapr/dapr --namespace dapr-system --set global.LogAsJSON=true
+helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
 ```
 
 ### Enable JSON formatted log for Dapr sidecars 

concepts/observability/metrics.md

@@ -1,30 +1,21 @@
 # Metrics
-Dapr exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain
-a greater understanding of how Dapr is behaving and to setup alerts for specific conditions.
+
+Dapr exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving and to setup alerts for specific conditions.
 
 ## Configuration
-The metrics endpoint is enabled by default, you can disable it by passing the command line argument
-`--enable-metrics=false` to daprd.
 
-The default metrics port is `9090`. This can be overridden by passing the command line argument
-`--metrics-port` to darpd.
+The metrics endpoint is enabled by default, you can disable it by passing the command line argument `--enable-metrics=false` to dapr system processes.
+
+The default metrics port is `9090`. This can be overridden by passing the command line argument `--metrics-port` to darpd.
 
 ## Metrics
-Dapr will expose metrics for Darp's HTTP pipeline, gPRC pipeline, Go runtime, metrics server and process.
-> Note: The gRPC metrics are all initialized with zero values and are therefore immediately visible on the 
-metrics endpoint. HTTP metrics are not zero initialized so will only become visible once a HTTP request has
-passed through the HTTP pipeline.
 
-## Prometheus
-To consume these metrics you'll need to install Prometheus or something compatible with scraping Prometheus
-metrics.
+Each Dapr system process emits Go runtime/process metrics by default and have their own metrics:
 
-For more detailed instructions on installing and configuring Prometheus to work with Dapr, please refer to the [Observe Metrics with Prometheus](../../howto/observe-metrics-with-prometheus/README.md) article.
+- [Dapr runtime metric list](https://github.com/dapr/dapr/blob/master/pkg/diagnostics/README.md)
 
 ## References
 
-* [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
-
-* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
-
-* [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
+* [Howto: Run Prometheus locally](../../howto/setup-monitoring-tools/observe-metrics-with-prometheus-locally.md)
+* [Howto: Set up Prometheus and Grafana for metrics](../../howto/setup-monitoring-tools/setup-prometheus-grafana.md)
+* [Howto: Set up Azure monitor to search logs and collect metrics for Dapr](../../howto/setup-monitoring-tools/setup-azure-monitor.md)

howto/README.md

@@ -51,10 +51,18 @@ For Actors How Tos see the SDK documentation
 * [Java Actors](https://github.com/dapr/java-sdk)
 
 ## Observerability
+
+### Metric and logs
+
+* [Set up Azure monitor to search logs and collect metrics for Dapr](./setup-monitoring-tools/setup-azure-monitor.md)
+* [Set up Fleuntd, Elastic search, and Kibana in Kubernetes](./setup-monitoring-tools/setup-fluentd-es-kibana.md)
+* [Set up Prometheus and Grafana for metrics](./setup-monitoring-tools/setup-prometheus-grafana.md)
+
 ### Distributed Tracing
 
 * [Diagnose your services with distributed tracing](./diagnose-with-tracing)
 
+
 ## Security
 ### Mutual Transport Layer Security (TLS)
 
@@ -87,4 +95,4 @@ For Actors How Tos see the SDK documentation
 
 ## Infrastructure integration
 
-* [Autoscale on Kubernetes using KEDA and Dapr bindings](./autoscale-with-keda)
+* [Autoscale on Kubernetes using KEDA and Dapr bindings](./autoscale-with-keda)

howto/setup-monitoring-tools/img/grafana-prometheus-dapr-server-url.png

@@ -1,4 +1,5 @@
-# Observe Metrics with Prometheus
+# Observe Metrics with Prometheus locally
+
 Dapr exposes a Prometheus metrics endpoint you can use to collect time-series
 data relating to the execution of the Dapr runtime itself.
 
@@ -55,36 +56,3 @@ docker run \
 `--net=host` ensures that the Prometheus instance will be able to connect to any Dapr instances running on the host machine. If you plan to run your Dapr apps in containers as well, you'll need to run them on a shared Docker network and update the configuration with the correct target address.
 
 Once Prometheus is running, you'll be able to visit its dashboard by visiting `http://localhost:8080`.
-
-## Setup Prometheus on Kubernetes
-
-Prometheus can be installed onto a Kubernetes cluster in a number of different ways. These are documented comprehensively [here](https://github.com/coreos/kube-prometheus).
-
-Once you have installed Prometheus on your Kubernetes cluster, you can add your Dapr sidecars' service address to your Prometheus configuration.
-
-The Dapr sidecar addresses can be fetched by using
-```
-$ kubectl get svc
-
-NAME          TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)
-nodeapp-dapr  ClusterIP  10.0.163.251  <none>       80/TCP,50001/TCP,9090/TCP
-...
-```
-
-You can then add the services DNS to your Prometheus configuration.
-```yaml
-...
-scrape_configs:
-  ...
-  - job_name: 'nodeapp-dapr'
-    metrics_path: /
-    static_configs:
-      - targets: ['nodeapp-dapr.<namespace>.svc.cluster.local:9090'] # Replace with Dapr metrics port if not default
-    ...
-```
-
-To see you Prometheus dashboard in Kubernetes, you'll need to port forward to your Prometheus pod and then hit the local endpoint `http://localhost:9090`.
-
-```
-kubectl port-forward <prometheus-pod-name> 9090:9090
-```

howto/setup-monitoring-tools/img/grafana-prometheus-server-url.png

@@ -59,7 +59,7 @@ kubectl apply -f ./azm-config.map.yaml
 1. Install Dapr with enabling JSON-formatted logs
 
 ```bash
-helm install dapr dapr/dapr --namespace dapr-system --set global.LogAsJSON=true
+helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
 ```
 
 2. Enable JSON formatted log in Dapr sidecar and add Prometheus annotations.

howto/observe-metrics-with-prometheus/README.md renamed to howto/setup-monitoring-tools/observe-metrics-with-prometheus-locally.md

@@ -86,7 +86,7 @@ fluentd-sdrld                 1/1     Running   0          14s
 1. Install Dapr with enabling JSON-formatted logs
 
 ```bash
-helm install dapr dapr/dapr --namespace dapr-system --set global.LogAsJSON=true
+helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
 ```
 
 2. Enable JSON formatted log in Dapr sidecar

howto/setup-monitoring-tools/setup-azure-monitor.md

@@ -30,10 +30,7 @@ helm install dapr-prom stable/prometheus -n dapr-monitoring
    If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command.
 
 ```bash
-helm install prometheus stable/prometheus -n dapr-monitoring
---set alertmanager.persistentVolume.enable=false
---set pushgateway.persistentVolume.enabled=false
---set server.persistentVolume.enabled=false
+helm install dapr-prom stable/prometheus -n dapr-monitoring --set alertmanager.persistentVolume.enable=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false
 ```
 
 3. Install Grafana
@@ -44,11 +41,20 @@ helm install grafana stable/grafana -n dapr-monitoring
 
    If you are minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command.
 
-   ```bash
+```bash
 helm install grafana stable/grafana -n dapr-monitoring --set persistence.enabled=false
 ```
 
-4. Validation
+4. Retrieve admin password for Grafana Login
+
+> Note: remove `%` character from the password that this command returns. The admin password is `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1`.
+
+```
+kubernetes get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode
+cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%
+```
+
+5. Validation
 
 Ensure Prometheus and Grafana are running in your cluster.
 
@@ -80,15 +86,17 @@ Handling connection for 8080
 
 2. Browse `http://localhost:8080`
 
-3. Click Configuration Settings -> Data Sources
+3. Login with admin and password
+
+4. Click Configuration Settings -> Data Sources
 
       ![data source](./img/grafana-datasources.png)
 
-4. Add Prometheus as a data soruce.
+5. Add Prometheus as a data soruce.
 
       ![add data source](./img/grafana-datasources.png)
 
-5. Enter Promethesus server address in your cluster.
+6. Enter Promethesus server address in your cluster.
 
 You can get the prometheus server address by running following command.
 
@@ -107,30 +115,35 @@ kibana-kibana                        ClusterIP   10.0.188.224   <none>        56
 
 ```
 
-In this set up tutorial, the server is `dapr-prom-prometheus-server`.
+In this howto, the server is `dapr-prom-prometheus-server`.
 
-So you need to provide `http://dapr-prom-prometheus-server.dapr-monitoring` in the URL field.
+So you need to set up Prometheus data source with the below settings:
 
-![prometheus server](./img/grafana-prometheus-server-url.png)
+- Name: `Dapr`
+- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring`
+- Default: On
 
-6. Click Save & Test button to verify that connected succeeded.
+![prometheus server](./img/grafana-prometheus-dapr-server-url.png)
 
-7. Import Dapr dashboards.
+7. Click `Save & Test` button to verify that connected succeeded.
 
-You can now import built-in [Grafana dashboard templates](https://github.com/dapr/docs/tree/master/monitoring/grafana/dashboards). 
+8. Import Dapr dashboards.
 
-Refer [here](https://github.com/dapr/docs/tree/master/monitoring/grafana) for details.
+You can now import built-in [Grafana dashboard templates](../../reference/dashboard/README.md).
+
+Refer [here](../../reference/dashboard/README.md) for details.
 
 ![upload json](./img/grafana-uploadjson.png)
 
-You can find screenshots of Dapr dashboards [here](https://github.com/dapr/docs/tree/master/monitoring/grafana/img).
+9. Find the dashboard that you imported and enjoy!
+
+![upload json](../../reference/dashboard/img/system-service-dashboard.png)
+
+You can find more screenshots of Dapr dashboards [here](../../reference/dashboard/img/).
 
 # References
 
 * [Prometheus Installation](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
-
 * [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
-
 * [Prometheus Kubernetes Operator](https://github.com/helm/charts/tree/master/stable/prometheus-operator)
-
-* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)
+* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)