Skip to content

Commit

Permalink
Update OSS-FORVM
Browse files Browse the repository at this point in the history 
update oss-graylog-forensics VM
  • Loading branch information
@timyehamazon
timyehamazon committed May 12, 2024
1 parent 3217616 commit 3be904d
Show file tree
Hide file tree
Showing 5 changed files with 179 additions and 28 deletions.
2 changes: 2 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

.DS_Store
113 changes: 113 additions & 0 deletions OSSVM_BUILD.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
oss-graylog-forensics Virtual Machine How To Build By YourSelf
=======

Build on debian 12

apt-get update && apt-get -y install lsb-release ca-certificates curl gnupg2
curl -o- https://artifacts.opensearch.org/publickeys/opensearch.pgp | sudo gpg --dearmor --batch --yes -o /usr/share/keyrings/opensearch-keyring
echo "deb [signed-by=/usr/share/keyrings/opensearch-keyring] https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list

export OPENSEARCH_INITIAL_ADMIN_PASSWORD=yourpassword
apt install opensearch

sudo systemctl daemon-reload
sudo systemctl enable opensearch.service

/etc/opensearch/opensearch.yml

```
cluster.name: oss-graylog-awsforensics
plugins.security.disabled: true
assistant.chat.enabled: true
```

wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.13.0/opensearch-dashboards-2.13.0-linux-x64.deb
sudo dpkg -i opensearch-dashboards-2.13.0-linux-x64.deb
sudo systemctl enable opensearch-dashboards.service

OpenSearch Dashboards remove plugins
```
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards --allow-root
```

Remove security related in opensearch-dashboard.yml

sudo systemctl start opensearch.service
sudo systemctl start opensearch-dashboards.service

Install graylog Open
```
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' >> sudo /etc/sysctl.conf
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg \
--dearmor
echo "deb [ signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] http://repo.mongodb.org/apt/debian bullseye/mongodb-org/7.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt update
sudo wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb
sudo dpkg -i libssl1.1_1.1.1f-1ubuntu2_amd64.deb
sudo apt-get install -y mongodb-org=7.0.7 mongodb-org-database=7.0.7
sudo systemctl start mongod && sudo systemctl enable mongod
wget https://packages.graylog2.org/repo/packages/graylog-5.2-repository_latest.deb
sudo dpkg -i graylog-5.2-repository_latest.deb
sudo apt update
sudo apt install -y graylog-server pwgen
sudo sed -i "s/password_secret =.*$/password_secret = $(pwgen -N 1 -s 96)/g" /etc/graylog/server/server.conf
read -s -p "Enter Password: " PASSWORD && echo && PASSWORD=$(echo -n "$PASSWORD" | sha256sum | cut -d" " -f1) && sudo sed -i "s/root_password_sha2 =.*$/root_password_sha2 = $PASSWORD/g" /etc/graylog/server/server.conf
# here set awsforensics as password
sudo sed -i "s@#root_timezone = UTC@root_timezone = Asia/Taipei@g" /etc/graylog/server/server.conf
sudo sed -i "s@#http_bind_address = 127.0.0.1:9000@http_bind_address = 0.0.0.0:9000@g" /etc/graylog/server/server.conf
sudo sed -i "s/allow_highlighting = false/allow_highlighting = true/g" /etc/graylog/server/server.conf
sudo sed -i 's/#elasticsearch_hosts = http:\/\/node1:9200,http:\/\/user:password@node2:19200/elasticsearch_hosts = http:\/\/127.0.0.1:9200/g' /etc/graylog/server/server.conf
sudo systemctl daemon-reload
sudo systemctl restart graylog-server
sudo systemctl enable graylog-server
sudo tail -f /var/log/graylog-server/server.log
# To get password and login graylog
```

sudo apt install logstash, filebeat

sudo ./logstash-plugin install logstash-input-relp
sudo ./logstash-plugin install logstash-input-google_pubsub
sudo ./logstash-plugin install logstash-filter-tld
sudo ./logstash-plugin install logstash-filter-rest
sudo ./logstash-plugin install logstash-filter-json_encode
sudo ./logstash-plugin install logstash-output-opensearch

Copy geoip mmdb sof-elk to oss-graylog-forensics ( /usr/share/GeoIP )

Fix logstash config
Example : /etc/logstash/conf.d/9901-output-aws.conf

```
# SOF-ELK® Configuration File
# (C)2021 Lewes Technology Consulting, LLC
#
# This file contains outputs for AWS logs
output {
if [type] == "aws" {
opensearch {
index => "aws-%{+YYYY.MM}"
template => "/usr/local/sof-elk/lib/elasticsearch-aws-template.json"
template_name => "aws"
template_overwrite => true
ecs_compatibility => "disabled"
}
}
}
```

1 change: 1 addition & 0 deletions OSSVM_DOWNLOAD.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
https://drive.google.com/file/d/1T9pBWZd3JnTPU2I-sLVMQMBlSVkj81Q0/view?usp=sharing
47 changes: 19 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,44 +1,35 @@
# SOF-ELK® Configuration Files
<oss-graylog-forensics VM beta>

![SOF-ELK Logo](https://raw.githubusercontent.com/philhagen/sof-elk/main/lib/sof-elk_logo_sm.png)
Modify from SOF-ELK (Great thanks!)

This repository contains the configuration and support files for the SOF-ELK® VM Appliance.
Replace ElasticSearch with OpenSearch (Has free sigma rules functionality)

SOF-ELK® is a “big data analytics” platform focused on the typical needs of computer forensic investigators/analysts and information security operations personnel. The platform is a customized build of the open source Elastic stack, consisting of the Elasticsearch storage and search engine, Logstash ingest and enrichment system, Kibana dashboard frontend, and Elastic Beats log shipper (specifically filebeat). With a significant amount of customization and ongoing development, SOF-ELK® users can avoid the typically long and involved setup process the Elastic stack requires. Instead, they can simply download the pre-built and ready-to-use SOF-ELK® virtual appliance that consumes various source data types (numerous log types as well as NetFlow), parsing out the most critical data and visualizing it on several stock dashboards. Advanced users can build visualizations the suit their own investigative or operational requirements, optionally contributing those back to the primary code repository.
VM contain pre-installed SOF-ELK with opensearch(not elasticsearch) and graylog open

The SOF-ELK® platform was initially developed for SANS FOR572, Advanced Network Forensics and Analysis, and is now used in several other SANS courses, with additional course integrations being considered. Most importantly, the platform is also distributed as a free and open source resource for the community at large, without a specific course requirement or tie-in required to use it.
VM can be found on [](OSSVM_DOWNLOAD.md) ( better have 6GB ram and free 40GB disk )

More details about the pre-packaged VM are available here: <https://for572.com/sof-elk-readme>.
How to build by yourself [](OSSVM_BUILD.md)

## Branches
Conference talk material can be found here :

* `main`: This branch is considered suitable for widespread use, but should not be used in the FOR572 class itself. The classroom labs are version-locked, but work on this repository is ongoing.
* `public/*`: These branches will be tied to public releases of the VM, allowing version-locked content control after deployment.
* `class/*`: When a VM is prepared for distribution in a SANS course such as FOR572, a new sub-branch will be created under the "class" branch with a name corresponding to the VM version. (e.g. "`class/v20170629`").
* `develop`: This branch contains code that should be functional, but may break at times (and remain broken). Of course, we'll try to avoid that, but it should be clear that this is NOT a branch suitable for "real world" use.
* Other branches may be used for major version updates, etc. These will be merged to main when deployed for mainstream use.
<https://cybersec.ithome.com.tw/2024/en/speaker-page/1343>

## Using
====== Original SOF-ELK message

These files are only recommended to be used in the SOF-ELK VM distribution at this time. A great deal of system-level configuration and tie-in is required for them to be used. No support can be provided for the use of these files outside the SOF-ELK VM as distributed via the [readme](https://for572.com/sof-elk-readme).
https://github.com/philhagen/sof-elk/issues/301

## Contents by directory
[](SOFELK-README.md)

* `/configfiles/`: These files conatain parsing/tagging/formatting/etc logic for individual file types as well as output configuration.
* `/configfiles-UNSUPPORTED/`: These configuration files are either not ready for operational use, in testing, or otherwise staged/stashed.
* `/doc/`: Documentation. Always a work in progress.
* `/grok-patterns/`: Custom parsing patterns used by the files in the `/configfiles/` directory.
* `/kibana/`: These files define the Kibana dashboards and associated files for individual data types. These correspond with the parsing completed by the Logstash files in the `/configfiles/` directory, so they probably won't work on your own Logstash instance without some tweaking. To load these to the Kibana interface, run the `/supporting-scripts/load_all_dashboards.sh` script.
* `/lib/`: Supporting files, including elasticsearch mappings, YAML lookup files, and images.
* `/supporting-scripts/`: Numerous scripts and supporting files needed for the SOF-ELK VM to function. Any scripts that may be required for user functionality are symlinked to be in the `elk_user`'s `$PATH`.
# SOF-ELK® Configuration Files

## Questions/Bug Reports/etc
![SOF-ELK Logo](https://raw.githubusercontent.com/philhagen/sof-elk/main/lib/sof-elk_logo_sm.png)

This repository contains the configuration and support files for the SOF-ELK® VM Appliance.

All bugs and feature requests should be logged via the github issue tracker: <https://github.com/philhagen/sof-elk/issues/>.
SOF-ELK® is a “big data analytics” platform focused on the typical needs of computer forensic investigators/analysts and information security operations personnel. The platform is a customized build of the open source Elastic stack, consisting of the Elasticsearch storage and search engine, Logstash ingest and enrichment system, Kibana dashboard frontend, and Elastic Beats log shipper (specifically filebeat). With a significant amount of customization and ongoing development, SOF-ELK® users can avoid the typically long and involved setup process the Elastic stack requires. Instead, they can simply download the pre-built and ready-to-use SOF-ELK® virtual appliance that consumes various source data types (numerous log types as well as NetFlow), parsing out the most critical data and visualizing it on several stock dashboards. Advanced users can build visualizations the suit their own investigative or operational requirements, optionally contributing those back to the primary code repository.

Please see the pull request submission guidelines before starting any development work - this is in the [](PULLREQUESTS.md) file.
The SOF-ELK® platform was initially developed for SANS FOR572, Advanced Network Forensics and Analysis, and is now used in several other SANS courses, with additional course integrations being considered. Most importantly, the platform is also distributed as a free and open source resource for the community at large, without a specific course requirement or tie-in required to use it.

## Administrative Notifications/Disclaimers/Legal/Boring Stuff
More details about the pre-packaged VM are available here: <https://for572.com/sof-elk-readme>.

* Content of this repository are provided "as is" with no express or implied warranty for accuracy or accessibility.
* SOF-ELK® is a registered trademark of Lewes Technology Consulting, LLC. Content is copyrighted by its respective contributors. SOF-ELK logo is a wholly owned property of Lewes Technology Consulting, LLC and is used by permission.
[readme](https://for572.com/sof-elk-readme).
44 changes: 44 additions & 0 deletions SOFELK-README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# SOF-ELK® Configuration Files

![SOF-ELK Logo](https://raw.githubusercontent.com/philhagen/sof-elk/main/lib/sof-elk_logo_sm.png)

This repository contains the configuration and support files for the SOF-ELK® VM Appliance.

SOF-ELK® is a “big data analytics” platform focused on the typical needs of computer forensic investigators/analysts and information security operations personnel. The platform is a customized build of the open source Elastic stack, consisting of the Elasticsearch storage and search engine, Logstash ingest and enrichment system, Kibana dashboard frontend, and Elastic Beats log shipper (specifically filebeat). With a significant amount of customization and ongoing development, SOF-ELK® users can avoid the typically long and involved setup process the Elastic stack requires. Instead, they can simply download the pre-built and ready-to-use SOF-ELK® virtual appliance that consumes various source data types (numerous log types as well as NetFlow), parsing out the most critical data and visualizing it on several stock dashboards. Advanced users can build visualizations the suit their own investigative or operational requirements, optionally contributing those back to the primary code repository.

The SOF-ELK® platform was initially developed for SANS FOR572, Advanced Network Forensics and Analysis, and is now used in several other SANS courses, with additional course integrations being considered. Most importantly, the platform is also distributed as a free and open source resource for the community at large, without a specific course requirement or tie-in required to use it.

More details about the pre-packaged VM are available here: <https://for572.com/sof-elk-readme>.

## Branches

* `main`: This branch is considered suitable for widespread use, but should not be used in the FOR572 class itself. The classroom labs are version-locked, but work on this repository is ongoing.
* `public/*`: These branches will be tied to public releases of the VM, allowing version-locked content control after deployment.
* `class/*`: When a VM is prepared for distribution in a SANS course such as FOR572, a new sub-branch will be created under the "class" branch with a name corresponding to the VM version. (e.g. "`class/v20170629`").
* `develop`: This branch contains code that should be functional, but may break at times (and remain broken). Of course, we'll try to avoid that, but it should be clear that this is NOT a branch suitable for "real world" use.
* Other branches may be used for major version updates, etc. These will be merged to main when deployed for mainstream use.

## Using

These files are only recommended to be used in the SOF-ELK VM distribution at this time. A great deal of system-level configuration and tie-in is required for them to be used. No support can be provided for the use of these files outside the SOF-ELK VM as distributed via the [readme](https://for572.com/sof-elk-readme).

## Contents by directory

* `/configfiles/`: These files conatain parsing/tagging/formatting/etc logic for individual file types as well as output configuration.
* `/configfiles-UNSUPPORTED/`: These configuration files are either not ready for operational use, in testing, or otherwise staged/stashed.
* `/doc/`: Documentation. Always a work in progress.
* `/grok-patterns/`: Custom parsing patterns used by the files in the `/configfiles/` directory.
* `/kibana/`: These files define the Kibana dashboards and associated files for individual data types. These correspond with the parsing completed by the Logstash files in the `/configfiles/` directory, so they probably won't work on your own Logstash instance without some tweaking. To load these to the Kibana interface, run the `/supporting-scripts/load_all_dashboards.sh` script.
* `/lib/`: Supporting files, including elasticsearch mappings, YAML lookup files, and images.
* `/supporting-scripts/`: Numerous scripts and supporting files needed for the SOF-ELK VM to function. Any scripts that may be required for user functionality are symlinked to be in the `elk_user`'s `$PATH`.

## Questions/Bug Reports/etc

All bugs and feature requests should be logged via the github issue tracker: <https://github.com/philhagen/sof-elk/issues/>.

Please see the pull request submission guidelines before starting any development work - this is in the [](PULLREQUESTS.md) file.

## Administrative Notifications/Disclaimers/Legal/Boring Stuff

* Content of this repository are provided "as is" with no express or implied warranty for accuracy or accessibility.
* SOF-ELK® is a registered trademark of Lewes Technology Consulting, LLC. Content is copyrighted by its respective contributors. SOF-ELK logo is a wholly owned property of Lewes Technology Consulting, LLC and is used by permission.

0 comments on commit 3be904d

Please sign in to comment.