Support Fedora SELinux configuration (was: Permission denied on pem certificate of service)

[sayden]

Hi folks,

I had to install the entire OS from scratch. When using elastic-package v0.58.1 to use stack up -v I get the same error all the time. The following log uses v0.56.0 after downgrading to see if it helps, but the error logs is the same:

➜  integrations git:(aws/remove-lm/ebs) ✗ elastic-package stack up -v
2022/08/01 14:42:28 DEBUG Enable verbose logging
2022/08/01 14:42:28  INFO New version is available - v0.58.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.58.1
Boot up the Elastic stack
Using profile /home/mcastro/.elastic-package/profiles/default.
Remember to load stack environment variables using 'eval "$(elastic-package stack shellinit)"'.
2022/08/01 14:42:28 DEBUG running command: /usr/bin/docker-compose version --short
2022/08/01 14:42:28 DEBUG Determined Docker Compose version: <nil>, the tool will use Compose V1
2022/08/01 14:42:28 DEBUG running command: /usr/bin/docker-compose -f /home/mcastro/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack config
Elasticsearch host: https://127.0.0.1:9200
Kibana host: https://127.0.0.1:5601
Username: elastic
Password: changeme
Custom build packages directory found: /home/mcastro/elastic/integrations/build/packages
Packages from the following directories will be loaded into the package-registry:
- built-in packages (package-storage:snapshot Docker image)
- /home/mcastro/elastic/integrations/build/packages
2022/08/01 14:42:29 DEBUG running command: /usr/bin/docker-compose version --short
2022/08/01 14:42:29 DEBUG Determined Docker Compose version: <nil>, the tool will use Compose V1
2022/08/01 14:42:29 DEBUG running command: /usr/bin/docker-compose -f /home/mcastro/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build
elasticsearch uses an image, skipping
elasticsearch_is_ready uses an image, skipping
kibana uses an image, skipping
kibana_is_ready uses an image, skipping
package-registry_is_ready uses an image, skipping
fleet-server uses an image, skipping
fleet-server_is_ready uses an image, skipping
elastic-agent uses an image, skipping
elastic-agent_is_ready uses an image, skipping
Building package-registry
Sending build context to Docker daemon  13.72MB
Step 1/4 : FROM docker.elastic.co/package-registry/distribution:snapshot
 ---> dc05f9c0a06e
Step 2/4 : ARG PROFILE
 ---> Using cache
 ---> a3dbc9c37cca
Step 3/4 : COPY profiles/${PROFILE}/stack/package-registry.config.yml /package-registry/config.yml
 ---> Using cache
 ---> bf8e01fecadd
Step 4/4 : COPY stack/development/ /packages/development
 ---> Using cache
 ---> 21644caadac4
Successfully built 21644caadac4
Successfully tagged elastic-package-stack_package-registry:latest
2022/08/01 14:42:29 DEBUG running command: /usr/bin/docker-compose version --short
2022/08/01 14:42:30 DEBUG Determined Docker Compose version: <nil>, the tool will use Compose V1
2022/08/01 14:42:30 DEBUG running command: /usr/bin/docker-compose -f /home/mcastro/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack up
Creating network "elastic-package-stack_default" with the default driver
Creating elastic-package-stack_package-registry_1       ... done
Creating elastic-package-stack_elasticsearch_1    ... done
Creating elastic-package-stack_elasticsearch_is_ready_1 ... done

ERROR: for kibana  Container "bd1af16365b2" is unhealthy.

ERROR: for package-registry_is_ready  Container "7414f487b3bd" is unhealthy.
ERROR: Encountered errors while bringing up the project.
Error: booting up the stack failed: running docker-compose failed: running command failed: running Docker Compose up command failed: exit status 1

• 2 containers failed, package-registry and kibana. Here are the logs of package-registry_is_ready:

➜  integrations git:(aws/remove-lm/ebs) ✗ docker logs 7414f487b3bd
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":277},"message":"Packages paths: /packages/development, /packages/production, /packages/staging, /packages/snapshot","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":278},"message":"Cache time for /: 10s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":279},"message":"Cache time for /index.json: 10s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":280},"message":"Cache time for /search: 10m0s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":281},"message":"Cache time for /categories: 10m0s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":282},"message":"Cache time for all others: 10m0s","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.399Z","log.origin":{"file.name":"package-registry/main.go","file.line":113},"message":"Package registry started","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:35.401Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/development","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:36.350Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/production","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:36.389Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/staging","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:36.525Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/snapshot","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:36.525Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/development","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:36.527Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/production","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:41.831Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/staging","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:42.065Z","log.origin":{"file.name":"packages/packages.go","file.line":229},"message":"Searching packages in /packages/snapshot","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:42.930Z","log.origin":{"file.name":"package-registry/main.go","file.line":300},"message":"1728 package manifests loaded","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-01T12:42:42.930Z","log.origin":{"file.name":"package-registry/main.go","file.line":169},"message":"Starting http metrics in 0.0.0.0:9000","ecs.version":"1.6.0"}
{"log.level":"fatal","@timestamp":"2022-08-01T12:42:42.930Z","log.origin":{"file.name":"package-registry/main.go","file.line":122},"message":"error occurred while serving","error":{"message":"open /etc/ssl/package-registry/cert.pem: permission denied"},"ecs.version":"1.6.0"}

• And the error of kibana container:

➜  integrations git:(aws/remove-lm/ebs) ✗ docker logs bd1af16365b2
Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from /usr/share/elasticsearch/config/elasticsearch.yml
	at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:128)
	at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:52)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.Command.main(Command.java:50)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/elasticsearch.yml
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
	at java.base/java.nio.file.Files.size(Files.java:2468)
	at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:91)
	... 8 more

Things I have tried:

• Remove everything from ~/.elastic-package
• Download v0.56.0 instead v0.58.1 (after removing like in the point above)
• elastic-package stack down and up again
• eval $(elastic-package stack shellinit) prior all the previous stuff

Other maybe not-so-useful information:

• Branch in integrations is main on commit 70d411b68ff3d49818cb25d9fdecd5bddf8dd83d

[sayden]

Aaaand 30 minutes after opening the issue the "light" came to me and I recall that small thing I always must do when I do a fresh Fedora installation:

Disable SELinux

Maybe it is not a solution that works for everybody but leaving it here for reference.

Sorry for the noise, folks! 😄

[jsoriano]

Reopening, if we have some time I think that it'd be nice to support SELinux, or at least document possible workarounds that don't imply to disable it completely.

Thanks for reporting!