Skip to content

Commit

Permalink
docs: postgresql: move installation up as first step
Browse files Browse the repository at this point in the history
  • Loading branch information
sebix committed Jan 8, 2025
1 parent 5b4d607 commit 803f9f6
Showing 1 changed file with 48 additions and 50 deletions.
98 changes: 48 additions & 50 deletions docs/admin/database/postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,23 @@ The EventDB is a database (usually [PostgreSQL](https://postgresql.org/)) that
gets filled with with data from IntelMQ using the SQL Output
Bot.

## intelmq_psql_initdb
## Setup

You have two basic choices to run PostgreSQL:

1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right
connection parameters to each psql or client call.

Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.

### PostgreSQL Version

Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [[1]](https://www.postgresql.org/support/versioning/).

If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone [formatting string](https://www.postgresql.org/docs/9.4/static/images/functions-formatting.html) "OF" for date-times and the [GiST index for the CIDR type](https://www.postgresql.org/docs/9.4/static/images/release-9-4.html#AEN120769). This may be useful depending on how you plan to use the events that this bot writes into the database.

### intelmq_psql_initdb

IntelMQ comes with the `intelmq_psql_initdb` command line tool designed to help with creating the
EventDB. It creates in the first line:
Expand All @@ -31,6 +47,36 @@ All elements of the generated SQL file can be adapted and extended before runnin
Be aware that if you create tables using another DB user that is used later by the output bot, you may need to adjust ownership or privileges in the database. If you have problems with database permissions,
refer to `PostgreSQL documentation <https://www.postgresql.org/docs/current/ddl-priv.html>`.

### Installation

Use `intelmq_psql_initdb` to create initial SQL statements from `harmonization.conf`. The script will create the
required table layout and save it as `/tmp/initdb.sql`

You need a PostgreSQL database-user to own the result database. The recommendation is to use the name `intelmq`
. There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from
setting up the expert/certbund_contact bot.)

Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system
user `postgres`:

```bash
createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq
```

Create the new database:

```bash
createdb --encoding='utf-8' --owner=intelmq intelmq-events
```

(The encoding parameter should ensure the right encoding on platform where this is not the default.)

Now initialize it as database-user `intelmq` (in this example a network connection to localhost is used, so you would
get to test if the user `intelmq` can authenticate):

```bash
psql -h localhost intelmq-events intelmq </tmp/initdb.sql
```
## EventDB Utilities

Some scripts related to the EventDB are located in the
Expand Down Expand Up @@ -181,55 +227,7 @@ The last steps brings us several advantages:
The complete SQL script can be generated using the `intelmq_psql_initdb`. It does *not* cover step 2 to avoid accidental
data loss - you need to do this step manually.


### Other docs

You have two basic choices to run PostgreSQL:

1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right
connection parameters to each psql or client call.

Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.

**PostgreSQL Version**

Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [[1]](https://www.postgresql.org/support/versioning/).

If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone [formatting string](https://www.postgresql.org/docs/9.4/static/images/functions-formatting.html) "OF" for date-times and the [GiST index for the CIDR type](https://www.postgresql.org/docs/9.4/static/images/release-9-4.html#AEN120769). This may be useful depending on how you plan to use the events that this bot writes into the database.

**How to install**

Use `intelmq_psql_initdb` to create initial SQL statements from `harmonization.conf`. The script will create the
required table layout and save it as `/tmp/initdb.sql`

You need a PostgreSQL database-user to own the result database. The recommendation is to use the name `intelmq`
. There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from
setting up the expert/certbund_contact bot.)

Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system
user `postgres`:

```bash
createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq
```

Create the new database:

```bash
createdb --encoding='utf-8' --owner=intelmq intelmq-events
```

(The encoding parameter should ensure the right encoding on platform where this is not the default.)

Now initialize it as database-user `intelmq` (in this example a network connection to localhost is used, so you would
get to test if the user `intelmq` can authenticate):

```bash
psql -h localhost intelmq-events intelmq </tmp/initdb.sql
```

**PostgreSQL and null characters**
## PostgreSQL and null characters

While null characters (`0`, not SQL "NULL") in TEXT and JSON/JSONB fields are valid, data containing null characters can
cause troubles in some combinations of clients, servers and each settings. To prevent unhandled errors and data which
Expand Down

0 comments on commit 803f9f6

Please sign in to comment.