authors	state
Roman Tkachenko (roman@goteleport.com)	implemented

RFD 11 - Teleport Database Access (Preview)

What

This document discusses high-level design points, user experience and some implementation details of the Teleport Database Access feature.

Note: This document refers to an early preview of the Database Access feature and covers functionality that will be available in the initial release.

With Teleport Database Access users can:

Provide secure access to databases without exposing them over the public network through Teleport's reverse tunnel subsystem.
Control access to specific database instances as well as individual databases and database users through Teleport's RBAC model.
Track individual users' access to databases as well as query activity through Teleport's audit log.

Use cases

The feature is being developed with the following use-cases in mind.

Human access

Users should be able to access the databases connected to Teleport using regular database clients they normally use to connect directly such as CLI clients (psql, mysql, etc.) as well as graphical interfaces (pgAdmin, MySQL Workbench, etc.).

The use-case for this is to grant users access to a database in a transparent fashion, for example to let them do development in a test/stage environment or perform an emergency recovery on a production database instance using familiar tools.

Robot access

The feature should be automation friendly so existing CI systems can take advantage of it.

An example would be letting the tools like Ansible or Drone perform routine actions on a database such as migrations or backups and be able to audit it.

Programmatic access

Programmatic access - as in configuring an application to talk to a database server through Teleport proxy - should work automatically as long as it uses a driver that properly implements a particular database protocol and supports mutual TLS authentication.

However, it is not the primary use-case, at least for the initial release, since it comes with a number of additional concerns and considerations such as performance requirements for high-traffic applications, automatic failover and so on.

Scope

For the initial release we're focusing on supporting a single type of database, PostgreSQL, with protocol parsing.

Supported protocols:

PostgreSQL wire protocol version 3.0, implemented in PostgreSQL 7.4 and later.

Supported authentication models:

Client certificate with PostgreSQL instances deployed on premise.
AWS RDS auth token with AWS RDS and PostgreSQL-compatible Aurora.

Supported features:

Connecting to the database through the Teleport proxy, incl. trusted clusters support.
Limiting access to database server instances by labels with Teleport roles.
Limiting access to individual databases (within a particular database server instance) and database users.
Auditing of database connections and executed queries.

Deployment modes

Example configurations Teleport database service can be used in:

Single-process mode where auth, proxy and database services all run within the same Teleport process. Useful for development and testing.
Database service is started separately and connects back to the cluster control plane over SSH reverse tunnel, same as application proxy service.
Trusted clusters, database service in a leaf cluster can be used in a same way after trusted cluster relationship with root has been established.

Architecture

Architecturally Database Access is very similar to Application Access. Teleport gains a new mode of operation, "database service", similar to "ssh service", "application service" or a "kube service".

Database service runs inside the customer private network and proxies database client connections received from the Teleport proxy to the target database. It is also responsible for parsing of supported database protocols and authorizing requests coming from the proxy (i.e. granting/denying access to particular databases/database users).

              |‾‾‾‾‾‾‾|  reverse  |‾‾‾‾‾‾‾‾‾‾‾‾|          |‾‾‾‾‾‾‾‾‾‾|
psql -------> | proxy | <-------- | db service | -------> | postgres |
              |_______|   tunnel  |____________|          |__________|

The database client (such as psql) will talk to Teleport web proxy port (3080 by default) which will use multiplexing to detect the database protocol and dispatch to an appropriate service.

Authentication

In this model, there are 3 points where authentication needs to happen.

Database client <---> proxy

Mutual TLS. Database clients, such as psql, will use short-lived x509 certificates issued by Teleport in order to authenticate with the proxy.

The certificate metadata includes (via extensions) routing information about the target database which users log into using tsh db login command (see below for UX).

Proxy <---> database service

Mutual TLS over SSH reverse tunnel. The connection that is passed from proxy to a Teleport database service is upgraded to TLS as well in order to be able to pass identity information over the reverse tunnel.

In order to do that, proxy will generate an ephemeral certificate signed by the user authority and re-encode all identity and routing information extracted from the certificate supplied by the client into it.

Database service <---> database instance

Mutual TLS (onprem). When connecting to an onprem database instance, the database service will use a client certificate for authentication, which it will generate on the fly according to the database requirements (for example, PostgreSQL requires the database user name to be encoded in the certificate as common name).

This also means that the database server needs to be configured with Teleport's certificate authority and (in general case) use server certificate issued by Teleport which can be done using tctl auth sign command (see below for UX).

Teleport database service can also be optionally configured with a custom CA certificate in case the database server uses cert/key pair signed by user's CA.

Password auth (RDS/Aurora). Amazon RDS/Aurora don't support client cert authentication. Instead, they support IAM authentication so Teleport database service will generate a short-lived authentication token using RDS API that will be used as a password:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html

In order to be able to verify RDS/Aurora server certificate, users will need to download RDS root certificate and configure the database service to trust it:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

Configuration

Teleport database service

The following new configuration section is added to the Teleport file config:

# New global key housing the database service configuration.
db_service:
  # Enable or disable the database service.
  enabled: "yes"
  # List of the database this service is proxying.
  databases:
    # Database instance name, used to refer to an instance in CLI like tsh.
  - name: "postgres-prod"
    # Optional free-form verbose description of a database instance.
    description: "Production instance of PostgreSQL 13.0"
    # Database protocol, only "postgres" is supported initially.
    protocol: "postgres"
    # Database connection URI, the address should be reachable from the service.
    uri: "postgres.internal.example.com:5432"
    # Optional CA cert path, e.g. RDS/Aurora root cert or custom onprem CA.
    ca_cert_file: "/path/to/root.pem"
    # AWS specific configuration for RDS/Aurora databases.
    aws:
      # Optional AWS region RDS/Aurora database is running in.
      region: "us-east-1"
    # Static labels assigned to the database instance, used in RBAC.
    static_labels:
      env: "stage"
    # Dynamic labels assigned to the database instance, used in RBAC.
    dynamic_labels:
    - name: "time"
      command: ["date", "+%H:%M:%S"]
      period: "1m"

When connecting Teleport database service to the cluster, users can either use a static join token (of type db) from auth server config, or generate a new join token for the service:

$ tctl tokens add --type=db

RDS/Aurora

When configuring Teleport database service with RDS/Aurora databases, the following fields need to be set:

db_service:
  enabled: "yes"
  databases:
  - name: "postgres-rds"
    protocol: "postgres"
    uri: "postgres-rds.xxx.us-east-1.rds.amazonaws.com:5432"
    ca_cert_file: "/opt/rds/rds-ca-2019-root.pem"
    aws:
      region: "us-east-1"

uri is the endpoint for a particular database from RDS control panel.
ca_cert_file is an optional path to the RDS root certificate.
aws.region is the region the database is deployed in.

If RDS root certificate is not provided explicitly, Teleport database service will attempt to download the correct version (based on the specified region the database is running in) automatically from AWS.

AWS credentials will be initialized using default credential provider chain used by Go SDK which looks in the environment variables, then shared credentials file and then falls back to IAM role. Refer to AWS documentation for more info.

Database instance

Onprem

To support mutual TLS auth, onprem database instance needs to be configured with Teleport's certificate authority and (optionally) cert/key pair issued by Teleport. The existing tctl auth sign command is used to produce these secrets:

$ tctl auth sign --format=db --host=localhost --out=server --ttl=8760h
$ ls
server.cas server.crt server.key

Note that the --host parameter should match the hostname of the endpoint the database will be connected at.

The certificate is signed by Teleport host authority which is intended for machine-to-machine communication.

The generated secrets are then used to configure the database. For example, for PostgreSQL:

$ cat postgresql.conf | grep ssl
ssl = on
ssl_ca_file = '/path/to/server.cas'
ssl_cert_file = '/path/to/server.crt'
ssl_key_file = '/path/to/server.key'

Database server may be configured with cert/key pair signed by a user's custom CA (for example, their organization's) in which case they would still need to supply Teleport's host CA so the server can authenticate Teleport client:

ssl_ca_file = '/path/to/server.cas'
ssl_cert_file = '/user/custom/server.pem'
ssl_key_file = '/user/custom/server.key'

In this case, Teleport database service must be configured with the appropriate CA certificate:

db_service:
  enabled: "yes"
  databases:
  - name: "postgres"
    ...
    ca_cert_file: "/user/custom/ca.pem"

In addition, PostgreSQL access configuration file should be configured to require client certificate authentication:

$ cat pg_hba.conf
# TYPE  DATABASE        USER            ADDRESS                 METHOD
hostssl all             all             ::/0                    cert
hostssl all             all             0.0.0.0/0               cert

RDS/Aurora

There are a few things that need to be done in order to configure RDS/Aurora database to support IAM authentication.

Enable IAM authentication when provisioning a new database, or on an existing one. By default, only password authentication is enabled.

More info:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Enabling.html

Configure IAM policy in order to allow the user Teleport database service will use to connect to the database instance with the auth token.

For example, to allow Teleport database service IAM user to connect to a particular RDS/Aurora instance as users alice or bob, the following policy may be defined:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "rds-db:connect",
            "Resource": [
                "arn:aws:rds-db:us-west-1:1234567890:dbuser:db-ABCDEFGHIJKLMNOP/alice",
                "arn:aws:rds-db:us-west-1:1234567890:dbuser:db-ABCDEFGHIJKLMNOP/bob"
            ]
        }
    ]
}

The resource definition also supports wildcards, so the policy can allow Teleport to use any database username, in which case the access will be enforced solely by Teleport's RBAC engine:

"Resource": [
  "arn:aws:rds-db:us-west-1:1234567890:dbuser:db-ABCDEFGHIJKLMNOP/*"
]

More info:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.IAMPolicy.html

Grant rds_iam role to each database user Teleport users will log in as.

For example, for PostgreSQL:

CREATE USER alice;
GRANT rds_iam TO alice;

More info:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.DBAccounts.html

RBAC

Teleport role resource gets 3 new fields that allow to control access to database instances as well as individual databases and database users.

kind: role
version: v3
metadata:
  name: developer
spec:
  allow:
    # Label selectors for database instances this role has access to.
    db_labels:
      environment: ["dev", "stage"]
    # Database names (within a database instance) this role has access to.
    # Note: this is not the same as the "name" field in "db_service".
    db_names: ["main", "metrics", "postgres"]
    # Database users this role can log in as.
    db_users: ["alice", "bob"]

Restricting access to specific database names and users may not be supported for all databases we will eventually support, only for those we will be doing protocol parsing for.

The new db_names and db_users rule properties support templating variables, similar to the existing fields, that allow to propagate them from the identity provider and between trusted clusters:

spec:
  allow:
    db_names: ["{{internal.db_names}}", "{{external.xxx}}"]
    db_users: ["{{internal.db_users}}", "{{external.yyy}}"]

For enterprise connectors, OIDC and SAML, the role mapping is sufficient to make use of these template variables. For the open-source Github connector we will extend teams_to_logins mapping:

spec:
  teams_to_logins:
  - organization: octocats
    team: admins
    db_names: ["test"]
    db_users: ["alice"]

The open-source tctl users add command will also be extended with additional flags that set allowed database names and users in the user traits:

$ tctl users add --db-names=main,metrics --db-users=postgres,alice alice

CLI

To connect to a database instance, a user first must login using regular tsh login command:

$ tsh login

Once logged in, they can see all available databases using the following command:

$ tsh db ls
Name            Description     Labels
--------------- --------------- ---------
postgres        PostgreSQL 13.0 env=dev
postgres-rds    PostgreSQL 12.4 env=stage
postgres-aurora PostgreSQL 11.6 env=prod

The list will only show databases the logged in user has access to (as per the database labels). For example, developer role shown above won't be able to see the Aurora database instance:

$ tsh db ls
Name         Description     Labels
------------ --------------- ---------
postgres     PostgreSQL 13.0 env=dev
postgres-rds PostgreSQL 12.4 env=stage

To log into a specific database instance, a user executes tsh db login command. This will retrieve the certificate with encoded database information in it:

$ tsh db login postgres

The tsh status command shows the name of the database a user is logged into:

$ tsh status
> Profile URL:        https://127.0.0.1:3080
  Logged in as:       alice
  Roles:              db-developer*
  ...
  Databases:          postgres
  ...

User can be logged into multiple databases simultaneously:

$ tsh db login db1
$ tsh db login db3
$ tsh db ls
Name  Description Labels
----- ----------- ---------
> db1 Database 1  env=dev
db2   Database 2  env=stage
> db3 Database 3  env=prod

To log out of a particular database (i.e. remove certificate from key store):

$ tsh db logout db1

Selecting databases and database user

The tsh db login command also prints a footer explaining how to connect to the database:

Connection information for "postgres" has been saved.
You can connect to the database using the following command:

  $ psql "service=root-postgres user=<user> dbname=<dbname>"

Or configure environment variables and use regular CLI flags:

  $ eval $(tsh db env)
  $ psql -U <user> <database>

After a successful login, users can use any database user and database to connect to, as long as it is allowed by the RBAC rules. For example:

$ tsh db login postgres-prod && eval $(tsh db env)
$ psql -U alice mydb     # allowed by role so will connect
$ psql -U bob mydb       # allowed by role so will connect
$ psql -U charlie mydb   # not allowed by role so will deny access

Default database user and name can also be provided to the tsh db login command, in which case then won't be required when using psql or other client:

$ tsh db login --db-user=alice --db-name=mydb postgres-prod
$ psql "service=postgres-prod"
$ eval $(tsh db env)
$ psql

Users can still choose to specify a user/database explicitly when connecting, for example to use a different database user or connect to a different database (subject to RBAC checks).

Connecting to PostgreSQL / service file

After logging into a PostgreSQL database, tsh configures an appropriate section in the connection service file which is located at ~/.pg_service.conf.

The service file has ini format and can contain multiple sections where each section defines connection parameters for a particular "service". PostgreSQL clients can refer to a particular service using service connection string parameter which all libpq-based clients should recognize (e.g. psql and pgAdmin do).

The section name has the format ${TELEPORT_CLUSTER}-${DATABASE_SERVICE} to avoid conflicts in situations where multiple trusted clusters have database services with the same name.

If a service file is already present, a new section will be added or existing section with the same name overwritten.

An added section may look like this:

[root-postgres-prod]
host=root.example.com
port=3080
sslmode=verify-full
sslrootcert=/home/user/.tsh/keys/root.example.com/certs.pem
sslcert=/home/user/.tsh/keys/root.example.com/alice-db/root/postgres-prod-x509.pem
sslkey=/home/user/.tsh/keys/root.example.com/alice

Users can then connect to the database using the following command:

$ psql "service=root-postgres-prod user=alice dbname=metrics"

Alternatively, tsh can output a set of environment variables supported by PostgreSQL clients for users to set in their session:

$ tsh db env
export PGHOST=root.example.com
export PGPORT=3080
export PGSSLMODE=verify-full
export PGSSLROOTCERT=/home/user/.tsh/keys/root.example.com/certs.pem
export PGSSLCERT=/home/user/.tsh/keys/root.example.com/alice-db/root/postgres-prod-x509.pem
export PGSSLKEY=/home/user/.tsh/keys/root.example.com/alice
$ eval $(tsh db env)
$ psql -U alice metrics

The output of tsh db env command is shell-specific, the shell is detected using the $SHELL environment variable value. The default output is compatible with shells like bash and zsh, in the future we'll add support for other shells like fish which has different syntax for setting environment variables.

pgAdmin

pgAdmin 4 is the Postgres GUI client. It has a client/server architecture and users interact with it by connecting to its server component via a browser.

To connect to a database instance using pgAdmin, the server must have access to credentials issued by tsh db login command.

pgAdmin recognizes Postgres service file so to add a new server connection users will need to specify:

[General / Name] Name used for identification purposes in the UI.
[Connection / Service] Postgres service name e.g. root-postgres-prod from examples above.
[Connection / Username] Username to use connecting to database.

All other fields such as hostname, port and SSL settings will be taken by pdAdmin from the service definition.

Audit events

The following new events are emitted to the Teleport audit log for database sessions.

Session start

db.session.start is emitted when a user has successfully connected to a database:

{
  "code": "TDB00I",
  "db_service": "postgres",
  "db_endpoint": "localhost:5432",
  "db_protocol": "postgres",
  "db_database": "postgres",
  "db_user": "postgres",
  "ei": 0,
  "event": "db.session.start",
  "namespace": "default",
  "server_id": "5603cd43-1172-4f1f-8e35-2bf74dfecf15",
  "sid": "427abfa3-fb1a-4a55-879c-77f5b6257cdb",
  "time": "2020-11-06T03:50:20.802Z",
  "uid": "f12b5199-5a1f-4e48-af4f-6980afffc48e",
  "user": "dev"
}

New fields are:

service: The name of Teleport database service handling the connection.
db_endpoint: The database instance address.
db_protocol: The database protocol.
db_database: The name of the database within DBMS a user is connecting to for databases with protocol parsing support.
db_user: The name of the database user a user is connecting as for databases with protocol parsing support.

Session end

db.session.end is emitted when a user has disconnected from the database:

{
  "code": "TDB01I",
  "db_service": "postgres",
  "db_endpoint": "localhost:5432",
  "db_protocol": "postgres",
  "db_database": "test",
  "db_user": "postgres",
  "ei": 0,
  "event": "db.session.end",
  "sid": "56335587-0157-4228-9556-aaa416741ef7",
  "time": "2020-11-06T03:50:20.802Z",
  "uid": "49e5f882-40ea-4bdb-b462-4cccf3536ada",
  "user": "dev"
}

Same new fields as in the db.session.start event.

Query

db.session.query is emitted when a user executes a database query:

{
  "code": "TDB02I",
  "db_service": "postgres",
  "db_endpoint": "localhost:5432",
  "db_protocol": "postgres",
  "db_database": "test",
  "db_query": "SELECT 1;",
  "db_user": "postgres",
  "ei": 0,
  "event": "db.session.query",
  "sid": "a53b0e1c-42f0-43ad-bbdd-9f3e07b54c05",
  "time": "2020-11-06T03:36:06.233Z",
  "uid": "7af4ea5d-ba1a-42d0-96a6-0c41b9c89bb0",
  "user": "dev"
}

Same new fields as in the db.session.start event, plus:

db_query: Full text of the executed query for databases with protocol parsing support.

CA rotation

As explained above, a database server is configured with Teleport's host certificate authority and certificate/key pair signed by the host CA as well, produced by tctl auth sign --type=db command.

This means that the database servers should be taken into consideration when performing Teleport's host CA rotation using tctl auth rotate command. See Teleport documentation for more information about certificate rotation.

Unlike Teleport nodes that automatically get reissued certificates signed by a new authority, we do not have control over database servers so a user must replace CA and keypair within the rotation grace period.

In addition, tctl auth rotate should detect that database access is configured and print instructions about rotating database secrets as well.

Future work

In future, instead of using host authority for signing certificates used by a database server, we can introduce another authority specifically for database access which would help decouple authority rotation for databases from the rest of the cluster.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0011-database-access.md

0011-database-access.md

RFD 11 - Teleport Database Access (Preview)

What

Use cases

Human access

Robot access

Programmatic access

Scope

Deployment modes

Architecture

Authentication

Database client <---> proxy

Proxy <---> database service

Database service <---> database instance

Configuration

Teleport database service

RDS/Aurora

Database instance

Onprem

RDS/Aurora

RBAC

CLI

Selecting databases and database user

Connecting to PostgreSQL / service file

pgAdmin

Audit events

Session start

Session end

Query

CA rotation

Files

0011-database-access.md

Latest commit

History

0011-database-access.md

File metadata and controls

RFD 11 - Teleport Database Access (Preview)

What

Use cases

Human access

Robot access

Programmatic access

Scope

Deployment modes

Architecture

Authentication

Database client <---> proxy

Proxy <---> database service

Database service <---> database instance

Configuration

Teleport database service

RDS/Aurora

Database instance

Onprem

RDS/Aurora

RBAC

CLI

Selecting databases and database user

Connecting to PostgreSQL / service file

pgAdmin

Audit events

Session start

Session end

Query

CA rotation