Skip to content
This repository has been archived by the owner on Dec 13, 2022. It is now read-only.

Commit

Permalink
Update documentation and minor fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
Alban MARGUET authored and tescure committed Jan 28, 2020
1 parent a06ee00 commit 720568b
Show file tree
Hide file tree
Showing 10 changed files with 115 additions and 85 deletions.
52 changes: 32 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,52 +3,64 @@
This project hosts files and links to components used by [Stormshield Visibility Center](https://www.stormshield.com/products/visibility-center).

## Installation instructions
* Make sure you have a fully functional Elastic stack running. If not, please refer to [Installing the Elastic Stack](https://www.elastic.co/guide/en/elastic-stack/6.8/installing-elastic-stack.html#installing-elastic-stack) instructions,
* Install [Syslog-ng](./syslog-ng)
* Install [Kibana index-pattern](./index-pattern),
* Install [Elasticsearch templates](./templates),
* Install [Logstash plugins](#plugins),
* Update [Logstash pipeline](./pipeline) configuration,
* Configure your Stormshield products to send logs to your _Syslog-ng_ instance
* UDP **514** or
* TCP **601**

* Make sure you have a fully functional Elastic stack running. If not, please refer to [Installing the Elastic Stack](https://www.elastic.co/guide/en/elastic-stack/6.8/installing-elastic-stack.html#installing-elastic-stack) instructions,
* Install [Syslog-ng](./syslog-ng)
* Install [Kibana index-pattern](./index-pattern),
* Install [Elasticsearch templates](./templates),
* Install [Logstash plugins](#plugins),
* Update [Logstash pipeline](./pipeline) configuration,
* Configure your Stormshield products to send logs to your _Syslog-ng_ instance
* UDP **514** or
* TCP **601**

### Docker

A ready to use Elastic Stack is also provided as a Docker container for testing
, [here](./docker).

## Logstash

### Plugins
* [logstash-filter-SNS](https://github.com/stormshield/logstash-filter-SNS)
* [logstash-filter-SDS](https://github.com/stormshield/logstash-filter-SDS)
* [logstash-filter-SES](https://github.com/stormshield/logstash-filter-SES)
* [logstash-filter-search-engine](https://github.com/stormshield/logstash-filter-search-engine)

* [logstash-filter-SNS](https://github.com/stormshield/logstash-filter-SNS)
* [logstash-filter-SDS](https://github.com/stormshield/logstash-filter-SDS)
* [logstash-filter-SES](https://github.com/stormshield/logstash-filter-SES)
* [logstash-filter-search-engine](https://github.com/stormshield/logstash-filter-search-engine)

### Pipeline configuration

List of [pipeline](./pipeline) configurations.

## Elasticsearch templates

List of [templates](./templates).

## Kibana index-pattern

List of [index-patterns](./index-pattern).

## Supported version
* Elasticsearch: *6.8.2*
* Kibana: *6.8.2*
* Logstash: *6.8.2*

* Elasticsearch: *6.8.2*
* Kibana: *6.8.2*
* Logstash: *6.8.2*

## Legal Disclaimer

Open source projects are made available and contributed to under licenses that include terms that, for the protection of contributors, make clear that the projects are offered “as-is”, without warranty, and disclaiming liability for damages resulting from using the projects. This guide is no different. The open content license it is offered under includes such terms.

Running an open source project, like any human endeavor, involves uncertainty and trade-offs. We hope this guide helps, but it may include mistakes, and can’t address every situation. If you have any questions about your project, we encourage you to do your own research, seek out experts, and discuss with your community. If you have any legal questions, you should consult with your own legal counsel before moving forward. If you’re at a company, talk to its legal team.

## Credit

SVC team:
* [Alban MARGUET](mailto:alban.marguet@stormshield.eu)
* [Laurent LEMKE](mailto:laurent.lemke@stormshield.eu)
* Nabil BENDAFI
* [Thomas ESCURE](mailto:thomas.escure@stormshield.eu)

* [Alban MARGUET](mailto:alban.marguet@stormshield.eu)
* [Laurent LEMKE](mailto:laurent.lemke@stormshield.eu)
* Nabil BENDAFI
* [Thomas ESCURE](mailto:thomas.escure@stormshield.eu)

## Contact

Labo SVC <[labo.svc@stormshield.eu](mailto:labo.svc@stormshield.eu)>
2 changes: 1 addition & 1 deletion docker/.env
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ELK_VERSION=5.6.7
ELK_VERSION=6.8.2
22 changes: 14 additions & 8 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,29 @@
This project hosts files and links to components used by [Stormshield Visibility Center](https://www.stormshield.com/products/visibility-center).

## Installation instructions
* Run following command to start Elastic Stack

* Run following command to start Elastic Stack

```bash
docker-compose up
```
* Open a web browser on http://127.0.0.1:5601
* username: **elastic**
* password: **changeme**

* Open a web browser on http://127.0.0.1:5601
* username: **elastic**
* password: **changeme**

## Supported version
* Elasticsearch : *5.6.7*
* Kibana: *5.6.7*
* Logstash: *5.6.7*

* Elasticsearch : *6.8.2*
* Kibana: *6.8.2*
* Logstash: *6.8.2*

## Legal Disclaimer

Open source projects are made available and contributed to under licenses that include terms that, for the protection of contributors, make clear that the projects are offered “as-is”, without warranty, and disclaiming liability for damages resulting from using the projects. This guide is no different. The open content license it is offered under includes such terms.

Running an open source project, like any human endeavor, involves uncertainty and trade-offs. We hope this guide helps, but it may include mistakes, and can’t address every situation. If you have any questions about your project, we encourage you to do your own research, seek out experts, and discuss with your community. If you have any legal questions, you should consult with your own legal counsel before moving forward. If you’re at a company, talk to its legal team.

## Credit
* [deviantony/docker-elk](https://github.com/deviantony/docker-elk) Github project

* [deviantony/docker-elk](https://github.com/deviantony/docker-elk) Github project
44 changes: 20 additions & 24 deletions index-pattern/README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,34 @@
# Kibana index-pattern

List of Kibana index-pattern, used for Stormshield product logs:
* stormshield-sns-fields-format.json [Network Security](https://www.stormshield.com/products-services/products/network-security/)
* stormshield-sdmc-fields-format.json [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/)
* stormshield-sds-fields-format.json [Data Enterprise](https://www.stormshield.com/products/enterprise)
* stormshield-ses-fields-format.json [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/)

* stormshield-sns-fields-format.json [Network Security](https://www.stormshield.com/products-services/products/network-security/)
* stormshield-sdmc-fields-format.json [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/)
* stormshield-sds-fields-format.json [Data Enterprise](https://www.stormshield.com/products/enterprise)
* stormshield-ses-fields-format.json [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/)

## Installation instructions

- For [Network Security](https://www.stormshield.com/products-services/products/network-security/) logs:
* For [Network Security](https://www.stormshield.com/products-services/products/network-security/) logs:

```bash
curl -XPOST -D- 'http://<your-kibana-server>:5601/api/saved_objects/index-pattern' \
-H 'Content-Type: application/json' \
-H 'kbn-version: 6.8.2' \
-H 'Content-Type: application/json' -d @stormshield-sns-fields-format.json
curl -XPOST --user elastic:changeme -D- 'http://<your-kibana-server>:5601/api/saved_objects/_bulk_create' -H 'kbn-version: 6.8.2' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d @stormshield-sns-fields-format.json
```
- For [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/) logs:

* For [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/) logs:

```bash
curl -XPOST -D- 'http://<your-kibana-server>:5601/api/saved_objects/index-pattern' \
-H 'Content-Type: application/json' \
-H 'kbn-version: 6.8.2' \
-H 'Content-Type: application/json' -d @stormshield-sdmc-fields-format.json
curl -XPOST --user elastic:changeme -D- 'http://<your-kibana-server>:5601/api/saved_objects/_bulk_create' -H 'kbn-version: 6.8.2' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d @stormshield-sdmc-fields-format.json
```
- For [Data Enterprise](https://www.stormshield.com/products/enterprise) logs:

* For [Data Enterprise](https://www.stormshield.com/products/enterprise) logs:

```bash
curl -XPOST -D- 'http://<your-kibana-server>:5601/api/saved_objects/index-pattern' \
-H 'Content-Type: application/json' \
-H 'kbn-version: 6.8.2' \
-H 'Content-Type: application/json' -d @stormshield-sds-fields-format.json
curl -XPOST --user elastic:changeme -D- 'http://<your-kibana-server>:5601/api/saved_objects/_bulk_create' -H 'kbn-version: 6.8.2' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d @stormshield-sds-fields-format.json
```
- For [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/) logs:

* For [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/) logs:

```bash
curl -XPOST -D- 'http://<your-kibana-server>:5601/api/saved_objects/index-pattern' \
-H 'Content-Type: application/json' \
-H 'kbn-version: 6.8.2' \
-H 'Content-Type: application/json' -d @stormshield-ses-fields-format.json
curl -XPOST --user elastic:changeme -D- 'http://<your-kibana-server>:5601/api/saved_objects/_bulk_create' -H 'kbn-version: 6.8.2' -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d @stormshield-ses-fields-format.json
```
31 changes: 17 additions & 14 deletions pipeline/README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,25 @@
# Logstash pipeline configuration files

List of Logstash pipeline configuration files, used for Stormshield product logs:
* 11-filter-standard.conf Standard filter that performs some pre-tasks
* 12-filter-sdmc.conf [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/)
* 13-filter-sds.conf [Data Enterprise](https://www.stormshield.com/products/enterprise)
* 14-filter-ses.conf [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/)
* 15-filter-sns.conf [Network Security](https://www.stormshield.com/products-services/products/network-security/)

* 11-filter-standard.conf Standard filter that performs some pre-tasks
* 12-filter-sdmc.conf [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/)
* 13-filter-sds.conf [Data Enterprise](https://www.stormshield.com/products/enterprise)
* 14-filter-ses.conf [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/)
* 15-filter-sns.conf [Network Security](https://www.stormshield.com/products-services/products/network-security/)

## Installation instructions
- Copy `01-input-syslog.conf` file in your _Logstash_ configuration path.
- Update the **port** on which _Logstash_ is listening for trafic, if needed.
- Copy `11-filter-standard.conf` and any _.conf_ files you want in your _Logstash_ configuration path
- Copy `21-output-elasticsearch.conf` file in your _Logstash_ configuration path.
- Replace **elasticsearch** host by your Elasticsearch instance hostname (eventually, update port number. Default _9200_), if needed.
- Replace or remove **user** and **password** as required
- Comment unwanted `if`/`else` sections
- Copy any _.template.json_ needed from [index-pattern](../index-pattern) to `/usr/share/logstash/templates/` (created directory if needed, which should belong to `logstash:logstash`)
- Install Logstash plugins:

* Copy `01-input-syslog.conf` file in your _Logstash_ configuration path.
* Update the **port** on which _Logstash_ is listening for traffic, if needed.
* Copy `11-filter-standard.conf` and any _.conf_ files you want in your _Logstash_ configuration path
* Copy `21-output-elasticsearch.conf` file in your _Logstash_ configuration path.
* Replace **elasticsearch** host by your Elasticsearch instance hostname (eventually, update port number. Default _9200_), if needed.
* Replace or remove **user** and **password** as required
* Comment unwanted `if`/`else` sections
* Copy any _.template.json_ needed from [index-pattern](../index-pattern) to `/usr/share/logstash/templates/` (created directory if needed, which should belong to `logstash:logstash`)
* Install Logstash plugins:

```bash
logstash-plugin install logstash-filter-SNS
logstash-plugin install logstash-filter-search-engine
Expand Down
2 changes: 1 addition & 1 deletion pipeline/files/01-input-syslog.conf
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
input {
syslog {
host => "127.0.0.1"
port => 5514
port => 5000
}
}
4 changes: 3 additions & 1 deletion pipeline/files/21-output-elasticsearch.conf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ output {
# stdout { codec => rubydebug }
elasticsearch {
document_type => "_doc"
hosts => "127.0.0.1"
hosts => "elasticsearch:9200"
user => "elastic"
password => "changeme"
index => "stormshield-%{internal_product}-%{type}-%{+YYYY.MM.dd}"
manage_template => false
}
Expand Down
4 changes: 3 additions & 1 deletion syslog-ng/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@
Stormshield products send logs using different RFC standards. This _Syslog-ng_ configuration file is used to address every cases, since [Logstash](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html) _syslog_ input plugin only supports [RFC3164](https://www.ietf.org/rfc/rfc3164.txt)

## Installation instructions
- Copy `syslog-stormshield-configuration.conf` file in your _Syslog-ng_ configuration path ( Default: _/etc/syslog-ng/conf.d/_)

* Modify the first line of the file `syslog-stormshield-configuration.conf`: _@version: 3.13_ with you own _Syslog-ng_ version if needed
* Copy `syslog-stormshield-configuration.conf` file in your _Syslog-ng_ configuration path ( Default: _/etc/syslog-ng/conf.d/_)
1 change: 1 addition & 0 deletions syslog-ng/syslog-stormshield-configuration.conf
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
@version: 3.13
@define allow-config-dups 1
@define MAX_CONNECTIONS 10
@define LOG_IW_SIZE 10000
Expand Down
38 changes: 23 additions & 15 deletions templates/README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,34 @@
# Elasticsearch templates

List of Elasticsearch templates, used for Stormshield product logs:
* sns.template.json [Network Security](https://www.stormshield.com/products-services/products/network-security/)
* sdmc.template.json [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/)
* sds.template.json [Data Enterprise](https://www.stormshield.com/products/enterprise)
* ses.template.json [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/)

* sns-*.template.json [Network Security](https://www.stormshield.com/products-services/products/network-security/)
* sdmc-*.template.json [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/)
* sds.template.json [Data Enterprise](https://www.stormshield.com/products/enterprise)
* ses-*.template.json [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/)

## Installation instructions

- For [Network Security](https://www.stormshield.com/products-services/products/network-security/) logs:
```bash
curl -XPUT http://<your-elasticsearch-server>:9200/_template/snslog -H 'Content-Type: application/json' -d @sns.template.json
* For [Network Security](https://www.stormshield.com/products-services/products/network-security/) logs:

```bash
find . -maxdepth 1 -name 'sns-*.template.json' -execdir bash -c 'file=${0#./}; curl --user elastic:changeme -XPUT http://<your-elasticsearch-server>:9200/_template/${file%.template.json} -H "Content-Type: application/json" -d @${file}' {} \;
```
- For [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/) logs:
```bash
curl -XPUT http://<your-elasticsearch-server>:9200/_template/sdmclog -H 'Content-Type: application/json' -d @sdmc.template.json

* For [Data For Cloud Mobility](https://www.stormshield.com/products/cloud-and-mobility/) logs:

```bash
find . -maxdepth 1 -name 'sdmc-*.template.json' -execdir bash -c 'file=${0#./}; curl --user elastic:changeme -XPUT http://<your-elasticsearch-server>:9200/_template/${file%.template.json} -H "Content-Type: application/json" -d @${file}' {} \;
```
- For [Data Enterprise](https://www.stormshield.com/products/enterprise) logs:
```bash
curl -XPUT http://<your-elasticsearch-server>:9200/_template/sdslog -H 'Content-Type: application/json' -d @sds.template.json

* For [Data Enterprise](https://www.stormshield.com/products/enterprise) logs:

```bash
curl --user elastic:changeme -XPUT http://<your-elasticsearch-server>:9200/_template/sds -H 'Content-Type: application/json' -d @sds.template.json
```
- For [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/) logs:

* For [Stormshield Endpoint Security](https://www.stormshield.com/products/stormshield-endpoint-security/) logs:

```bash
curl -XPUT http://<your-elasticsearch-server>:9200/_template/seslog -H 'Content-Type: application/json' -d @ses.template.json
find . -maxdepth 1 -name 'ses-*.template.json' -execdir bash -c 'file=${0#./}; curl --user elastic:changeme -XPUT http://<your-elasticsearch-server>:9200/_template/${file%.template.json} -H "Content-Type: application/json" -d @${file}' {} \;
```

0 comments on commit 720568b

Please sign in to comment.