Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Fetch Cilium Observe data to monitoring Cilium Service network traffic #12393

Merged
merged 11 commits into from
Jul 2, 2024
Merged

Support Fetch Cilium Observe data to monitoring Cilium Service network traffic #12393

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
412dbea
Support Cilium Fetcher
mrproliu Jun 19, 2024
a7cdbc4
Merge branch 'master' into cilium
mrproliu Jul 1, 2024
a9bec50
update DNS id
mrproliu Jul 1, 2024
877eb39
fix license
mrproliu Jul 1, 2024
6cb6c01
fix license
mrproliu Jul 1, 2024
8fb36ff
fix comments
mrproliu Jul 2, 2024
cd3591a
Merge branch 'master' into cilium
mrproliu Jul 2, 2024
8ce5000
update menu
mrproliu Jul 2, 2024
c2fab17
fix document
mrproliu Jul 2, 2024
9384329
Update docs/en/setup/backend/backend-k8s-monitoring-cilium.md
wu-sheng Jul 2, 2024
f6606dc
Update docs/en/setup/backend/backend-k8s-monitoring-cilium.md
wu-sheng Jul 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .licenserc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ header:
- '**/src/main/proto/jaeger/**'
- '**/src/main/proto/mixer/**'
- '**/src/main/proto/policy/**'
- '**/src/main/proto/cilium/**'
- '**/src/main/proto/prometheus/client_model/metrics.proto'
- '**/src/main/proto/protoc-gen-swagger/**'
- '**/src/main/proto/validate/validate.proto'
Expand Down
5 changes: 5 additions & 0 deletions dist-material/release-docs/LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,8 @@ The text of each license is the standard Apache 2.0 license.
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-graphql/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-graphql-protocol/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-grpc/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-grpc-protocol/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.linecorp.armeria/armeria-protobuf/1.27.3 Apache-2.0
https://mvnrepository.com/artifact/com.orbitz.consul/consul-client/1.5.3 Apache-2.0
https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp/3.14.9 Apache-2.0
Expand Down Expand Up @@ -295,6 +297,7 @@ The text of each license is the standard Apache 2.0 license.
https://mvnrepository.com/artifact/io.grpc/grpc-netty/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-protobuf/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-protobuf-lite/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-services/1.61.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-stub/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.grpc/grpc-util/1.63.0 Apache-2.0
https://mvnrepository.com/artifact/io.micrometer/micrometer-commons/1.12.2 Apache-2.0
Expand Down Expand Up @@ -371,6 +374,7 @@ The text of each license is the standard Apache 2.0 license.
https://mvnrepository.com/artifact/org.xerial.snappy/snappy-java/1.1.8.4 Apache-2.0
https://mvnrepository.com/artifact/org.yaml/snakeyaml/2.0 Apache-2.0
https://npmjs.com/package/typescript/v/4.7.4 4.7.4 Apache-2.0
https://github.com/cilium/cilium/tree/v1.15.6/api/v1 Apache-2.0

========================================================================
BSD-2-Clause licenses
Expand Down Expand Up @@ -544,6 +548,7 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/nanoid/v/3.3.7 3.3.7 MIT
https://mvnrepository.com/artifact/org.checkerframework/checker-qual/3.33.0 MIT
https://mvnrepository.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.23 MIT
https://mvnrepository.com/artifact/org.curioswitch.curiostack/protobuf-jackson/2.2.0 MIT
https://npmjs.com/package/pinia/v/2.0.28 2.0.28 MIT
https://npmjs.com/package/pinia/node_modules/vue-demi/v/0.13.11 0.13.11 MIT
https://npmjs.com/package/postcss/v/8.4.33 8.4.33 MIT
Expand Down
3 changes: 3 additions & 0 deletions dist-material/release-docs/LICENSE.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/{{ .Name }}/v/{{ .Version }} {{ .Version }} {{ .LicenseID }}
{{- end }}
{{- end }}
{{- if eq .LicenseID "Apache-2.0" }}
https://github.com/cilium/cilium/tree/v1.15.6/api/v1 Apache-2.0
{{- end }}
{{ end }}
=======================================================================
The zipkin-lens.jar dependency has more front-end dependencies in it and the front-end dependencies' licenses
Expand Down
7 changes: 6 additions & 1 deletion docs/en/changes/changes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,14 @@
* [Break Change] Update Nacos version to 2.3.2. Nacos 1.x server can't serve as cluster coordinator and configuration server.
* Support tracing trace query(SkyWalking and Zipkin) for debugging.
* Fix BanyanDB metrics query: used the wrong `Downsampling` type to find the schema.
* Support fetch cilium flow to monitoring network traffic between cilium services.
* Support `labelCount` function in the OAL engine.

#### UI

* Highlight search log keywords.
* Add Error URL in the browser log.
* Add a SolonMVC icon.
* Adding cilium icon and i18n for menu.

#### Documentation

Expand Down
5 changes: 5 additions & 0 deletions docs/en/concepts-and-designs/oal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@ In this case, see `p99`, `p95`, `p90`, `p75`, and `p50` of all incoming requests

In this case, the p99 value of all incoming requests. The parameter is precise to a latency at p99, such as in the above case, and 120ms and 124ms are considered to produce the same response time.

- `labelCount`. The count of the label value.
> drop_reason_count = from(CiliumService.*).filter(verdict == "dropped").labelCount(dropReason);

In this case, the count of the drop reason of each Cilium service.

## Metrics name
The metrics name for storage implementor, alarm and query modules. The type inference is supported by core.

Expand Down
104 changes: 103 additions & 1 deletion docs/en/concepts-and-designs/scope-definitions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -566,4 +566,106 @@ For `K8SEndpoint` and `K8SEndpointRelation`, they only have the following **prot
| componentId | The ID of component used in this call. | | string |
| destServiceName | The dest service name in kubernetes. | | string |
| destServiceName | The layer in kubernetes dest service. | | string |
| destEndpointName | The endpoint name detect in kubernetes dest service. | | string |
| destEndpointName | The endpoint name detect in kubernetes dest service. | | string |

### SCOPES with `Cilium` Prefix

All metrics starting with `Cilium` are derived from Cilium monitoring by Cilium Hubble.

#### Service, Service Instance and relations

For all `CiliumService`, `CiliumServiceInstance`, `CiliumServiceRelation` and `CiliumServiceInstanceRelation`, they all have the
following **L4**/**L7** metric contents.

| Name | Remarks | Group Key | Type |
|-----------------------|-----------------------------------------------------------------------------|-----------|--------|
| verdict | The metrics verdict from Flow. The value may be `forwarded` and `dropped`. | | string |
| type | The metrics type from Flow. The value may be `tcp`, `http`, `dns`, `kakfa`. | | string |
| direction | The metrics direction from Flow. The value may be `ingress` and `egress`. | | string |
| dropReason | When the verdict is `dropped`, the drop reason would be recorded. | | string |
| http.url | The URL of the HTTP request. | | string |
| http.code | The Response code of the HTTP response. | | int |
| http.protocol | The protocol of the HTTP request. | | string |
| http.method | The method of the HTTP request. | | string |
| kafka.errorCode | The error code of the Kafka request. | | int |
| kafka.errorCodeString | The error code explaination of the Kafka request. | | string |
| kafka.apiVersion | The API version of the Kafka request. | | string |
| kafka.apiKey | The API key of the Kafka request. | | string |
| kafka.correlationId | The correlation ID of the Kafka request. | | string |
| kafka.topic | The topic of the Kafka request. | | string |
| dns.domain | The domain of the DNS request. | | string |
| dns.queryType | The query type of the DNS request. | | string |
| dns.rcode | The response code of the DNS request. | | int |
| dns.recodeString | The response code explaination of the DNS request. | | string |
| dns.ttl | The TTL of the DNS request. | | int |
| dns.ipCount | The count of the IP addresses of the DNS responsed. | | int |
| duration | The duration(millisecond) of the L7 response. | | long |
| success | Is the response success of the L7 response. | | bool |

##### SCOPE `CiliumService`

| Name | Remarks | Group Key | Type |
|-------------|--------------------------------------------------------------------|-----------|--------|
| name | The service name in Cilium. | | string |
| layer | The layer in Cilium service. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |

##### SCOPE `CiliumServiceInstance`

| Name | Remarks | Group Key | Type |
|---------------------|--------------------------------------------------------------------|-----------|--------|
| serviceName | The service name in Cilium. | | string |
| serviceInstanceName | The pod name in Cilium. | | string |
| layer | The layer of Cilium service. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |

##### SCOPE `CiliumServiceRelation`

| Name | Remarks | Group Key | Type |
|-------------------|--------------------------------------------------------------------|-----------|--------|
| sourceServiceName | The source service name in Cilium. | | string |
| sourceLayer | The source layer service in Cilium. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |
| componentId | The ID of component used in this call. | | int |
| destServiceName | The dest service name in Cilium. | | string |
| destLayer | The dest layer service in Cilium. | | string |

##### SCOPE `CiliumServiceInstanceRelation`

| Name | Remarks | Group Key | Type |
|---------------------------|--------------------------------------------------------------------|-----------|--------|
| sourceServiceName | The source service name in Cilium. | | string |
| sourceServiceInstanceName | The source pod name in Cilium. | | string |
| sourceLayer | The source layer service in Cilium. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |
| componentId | The ID of component used in this call. | | int |
| destServiceName | The dest service name in Cilium. | | string |
| destServiceInstanceName | The dest pod name in Cilium. | | string |
| destLayer | The dest layer service in Cilium. | | string |

#### Endpoint and Endpoint Relation

For `CiliumEndpoint` and `CiliumEndpointRelation`, they have all the fields of **L4**/**L7** metric contents, but the `type` only would be `http`, `dns` or `kafka`.

##### SCOPE `CiliumEndpoint`

| Name | Remarks | Group Key | Type |
|--------------|---------------------------------------------------------|-----------|--------|
| serviceName | The service name in Cilium. | | string |
| layer | The layer in Cilium service. | | string |
| endpointName | The endpoint name detect in Cilium service. | | string |

##### SCOPE `CiliumEndpointRelation`

| Name | Remarks | Group Key | Type |
|--------------------|--------------------------------------------------------------------|-----------|--------|
| sourceServiceName | The source service name in Cilium. | | string |
| sourceLayer | The layer in Cilium source service. | | enum |
| sourceEndpointName | The endpoint name detect in Cilium source service. | | string |
| detectPoint | Where the relation is detected. The value may be client or server. | | enum |
| componentId | The ID of component used in this call. | | int |
| destServiceName | The dest service name in Cilium. | | string |
| destLayer | The layer in Cilium dest service. | | enum |
| destEndpointName | The endpoint name detect in Cilium dest service. | | string |


1 change: 1 addition & 0 deletions docs/en/guides/How-to-build.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,4 +84,5 @@ Refer to [Build docker image](../../../docker) for more details.
* `grpc-java` and `java` folders in **oap-server/exporter/target/generated-sources/protobuf**
* `grpc-java` and `java` folders in **oap-server/server-configuration/grpc-configuration-sync/target/generated-sources/protobuf**
* `grpc-java` and `java` folders in **oap-server/server-alarm-plugin/target/generated-sources/protobuf**
* `grpc-java` and `java` folders in **oap-server/server-fetcher-plugin/fetcher-proto/target/generated-sources/protobuf**
* `antlr4` folder in **oap-server/oal-grammar/target/generated-sources**
Loading
Loading