Skip to content

Commit c05533a

Browse files
committed
[DOCS][SPARK-677] Document secrets requirement: Mesos containerizer. (apache#281)
1 parent 8658312 commit c05533a

File tree

2 files changed

+12
-2
lines changed

2 files changed

+12
-2
lines changed

docs/kerberos.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ installation parameters, however does require the Spark Driver _and_ the Spark E
216216

217217
* The `keytab` containing the credentials for accessing the Kafka cluster.
218218

219+
--conf spark.mesos.containerizer=mesos # required for secrets
219220
--conf spark.mesos.driver.secret.names=<base64_encoded_keytab> # e.g. spark/__dcos_base64__kafka_keytab
220221
--conf spark.mesos.driver.secret.filenames=<keytab_file_name> # e.g. kafka.keytab
221222
--conf spark.mesos.executor.secret.names=<base64_encoded_keytab> # e.g. spark/__dcos_base64__kafka_keytab

docs/security.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,15 +60,22 @@ When you reference the `__dcos_base64__mysecret` secret in your service, the con
6060

6161
Once a secret has been added in the secret store,
6262
you can pass them to Spark with the `spark.mesos.<task-name>.secret.names` and
63-
`spark.mesos.<task-name>.secret.<filenames|envkeys>` confguration parameters where `<task-name>` is either `driver` or
63+
`spark.mesos.<task-name>.secret.<filenames|envkeys>` configuration parameters where `<task-name>` is either `driver` or
6464
`executor`. Specifying `filenames` or `envkeys` will materialize the secret as either a file-based secret or an
6565
environment variable. These configuration parameters take comma-separated lists that are "zipped" together to make the
6666
final secret file or environment variable. We recommend using file-based secrets whenever possible as they are more
6767
secure than environment variables.
68-
68+
69+
**NOTE**: Secrets are only supported for Mesos containerizer and not for the Docker containerizer.
70+
To use the Mesos containerizer, add this configuration:
71+
```
72+
--conf spark.mesos.containerizer=mesos
73+
```
74+
6975
For example to use a secret named `spark/my-secret-file` as a file in the driver _and_ the executors add these configuration
7076
parameters:
7177
```
78+
--conf spark.mesos.containerizer=mesos
7279
--conf spark.mesos.driver.secret.names=spark/my-secret-file
7380
--conf spark.mesos.driver.secret.filenames=target-secret-file
7481
--conf spark.mesos.executor.secret.names=spark/my-secret-file
@@ -78,6 +85,7 @@ this will put the contents of the secret `spark/my-secret-file` in a secure RAM-
7885
`target-secret-file` in the driver and executors sandboxes. If you want to use a secret as an environment variable (e.g.
7986
AWS credentials) you change the configurations to be the following:
8087
```
88+
--conf spark.mesos.containerizer=mesos
8189
--conf spark.mesos.driver.secret.names=/spark/my-aws-secret,/spark/my-aws-key
8290
--conf spark.mesos.driver.secret.envkeys=AWS_SECRET_ACCESS_KEY,AWS_ACCESS_KEY_ID
8391
```
@@ -88,6 +96,7 @@ This assumes that your secret access key is stored in a secret named `spark/my-a
8896
When using a combination of environment and file-based secrets there needs to be an equal number of sinks and secret
8997
sources (i.e. files and environment variables). For example
9098
```
99+
--conf spark.mesos.containerizer=mesos
91100
--conf spark.mesos.driver.secret.names=/spark/my-secret-file,/spark/my-secret-envvar
92101
--conf spark.mesos.driver.secret.filenames=target-secret-file,placeholder-file
93102
--conf spark.mesos.driver.secret.envkeys=PLACEHOLDER,SECRET_ENVVAR

0 commit comments

Comments
 (0)