Kubernetes deployment

[nadilas]

Hello guys,

I've been trying to convert the docker-compose to a kubernetes setup.

The issue I ran into is, that the config folder remains empty after the fusionauth-app container deploys. I presume upon the first startup the fusionauth.properties and keystore files would be either downloaded from created from the environment variables provided in the deplyoment file

      containers:
      - env:
        - name: DATABASE_PASSWORD
          value: <password>
        - name: DATABASE_ROOT_PASSWORD
          value: <password>
        - name: DATABASE_ROOT_USER
          value: postgres
        - name: DATABASE_URL
          value: jdbc:postgresql://fusionauth-db:5432/fusionauth
        - name: DATABASE_USER
          value: fusionauth
        - name: FUSIONAUTH_MEMORY
          value: 256M
        - name: FUSIONAUTH_SEARCH_SERVERS
          value: http://fusionauth-search:9200
        image: fusionauth/fusionauth-app:latest
        name: fusionauth
        ports:
        - containerPort: 9011

However the /usr/local/fusionauth/config is empty after startup.
If I manually create the properties file during initialization, we're one step closer. :)

What’s the condition for the “silent configuration” to be triggered?

[robotdan]

Hi @nadilas - thanks for your effort and contribution.

When we build the docker image, we download and extract the zip file for the fusionauth-app bundle. This process will create the config/fusionauth.properties etc.

It looks like you figured this out already, but the link in the Dockerfile on master is for the upcoming release. That was why the file does not exist yet. You can always find the latest available release on the release notes page. https://fusionauth.io/docs/v1/tech/release-notes/

If I manually create the properties file during initialization, we're one step closer. :)

You shouldn't need to manually create it. I would expect that using the image: fusionauth/fusionauth-app:latest would download the app image version 1.3.1. ( https://hub.docker.com/r/fusionauth/fusionauth-app/tags ). Which should already have the config directory set up in the image. Have you tried running docker run with the fusionauth-app image to see if you see the config directory there?

Other than that - I may have to try to recreate your symptom so I can debug it.

What’s the condition for the “silent configuration” to be triggered?

If the DATABASE_ROOT_USER and the DATABASE_URL are defined as environment variables during startup we will attempt to enter the silent configuration mode. This allows us to configure the schema, and create the search index without prompting the user. All other ENV values are optional as we'll try to default them - but you will probably want to define them if you can.

https://fusionauth.io/docs/v1/tech/installation-guide/docker#docker-services

I don't know enough about Kubernetes to help much here, but if it helps in the way you build out this configuration, there is a PR that I plan to integrate which has a better setup for environment variables and networking. #3

Let me know if you get any farther, if not I'll try to get setup to recreate your symptom so we can get to the bottom of it. Thanks for the help!

[nadilas]

Hi @robotdan,

great summary, thanks!
You are right, I am using fusionauth/fusionauth-app:latest, so it can only pull 1.3.1.
I have both DATABASE_ROOT_USER and DATABASE_URL defined as environment variables. When the container starts, it starts skips the setup and starts directly without creating thefusionauth.properties or the keystore files.

Maybe you find something obvious, what I missed: https://github.com/nadilas/fusionauth-k8s-infrastructure

I did a sanity check, running docker-compose up enters the silent setup mode and sets up both the db, the properties file and the keystore objects. So without kubernetes in the way, the fusionauth/fusionauth-app:latest image works fine.

Based on your summary, the I can only think that during the building of the docker image, the download cannot happen or the environment variables are not being picked up, which is wierd.

[nadilas]

Oh, I almost forgot, here's the log:

20-Jan-2019 16:33:21.285 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server}Setting property 'port' to '${fusionauth.management.port}' did not find a matching property.
20-Jan-2019 16:33:21.576 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'port' to '${fusionauth.http.port}' did not find a matching property.
20-Jan-2019 16:33:21.579 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'redirectPort' to '${fusionauth.https.port}' did not find a matching property.
20-Jan-2019 16:33:21.740 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'port' to '${fusionauth.https.port}' did not find a matching property.
20-Jan-2019 16:33:21.949 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'port' to '${fusionauth.ajp.port}' did not find a matching property.
20-Jan-2019 16:33:22.120 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
20-Jan-2019 16:33:22.562 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-auto-1"]
20-Jan-2019 16:33:22.610 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
20-Jan-2019 16:33:22.635 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio-auto-2"]
20-Jan-2019 16:33:23.511 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector[Connector[HTTP/1.1-auto-2]]
 org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-auto-2]]
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:549)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:632)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:655)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: /usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/keystore (Nosuch file or directory)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:116)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:87)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:225)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1086)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:268)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
        ... 13 more
Caused by: java.io.FileNotFoundException: /usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/keystore (No suchfile or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
        at org.apache.tomcat.util.file.ConfigFileLoader.getInputStream(ConfigFileLoader.java:96)
        at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:132)
        at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:204)
        at org.apache.tomcat.util.net.jsse.JSSEUtil.getKeyManagers(JSSEUtil.java:184)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)
        ... 20 more

20-Jan-2019 16:33:23.519 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-auto-3"]
20-Jan-2019 16:33:23.521 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
20-Jan-2019 16:33:23.523 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2463 ms
20-Jan-2019 16:33:23.540 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
20-Jan-2019 16:33:23.545 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.31
Jan 20, 2019 4:33:26 PM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete listof JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 20, 2019 4:33:29.008 PM ERROR org.primeframework.mvc.guice.GuiceBootstrap -

===================================================================================================

  Unable to start the server. Here's why:


[Error injecting constructor, java.lang.IllegalArgumentException: The configuration file [/usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/fusionauth.properties] doesn't exist.]
        -> [class java.lang.IllegalArgumentException] The configuration file [/usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/fusionauth.properties] doesn't exist.

===================================================================================================


Jan 20, 2019 4:33:29.041 PM ERROR org.primeframework.mvc.guice.GuiceBootstrap - Unable to start the server. Exception:

com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.IllegalArgumentException: The configuration file [/usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/fusionauth.properties] doesn't exist.
  at io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration.<init>(PropertiesFileFusionAuthConfiguration.java:35)
  at io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration.class(PropertiesFileFusionAuthConfiguration.java:21)
  while locating io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration
  at io.fusionauth.app.maintenance.guice.FusionAuthMaintenanceModeSilentConfigurationModule.configure(FusionAuthMaintenanceModeSilentConfigurationModule.java:30)
  while locating com.inversoft.configuration.InversoftConfiguration
    for the 1st parameter of com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService.downloadMySQLConnector(JDBCMaintenanceModeDatabaseService.java:67)

1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:178) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) ~[guice-4.2.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:87) ~[guice-4.2.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:69) ~[guice-4.2.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:59) ~[guice-4.2.0.jar:na]
        at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:58) ~[prime-mvc-1.11.1.jar:1.11.1]
        at com.inversoft.maintenance.servlet.MaintenanceModePrimeServletContextListener.contextInitialized(MaintenanceModePrimeServletContextListener.java:39) [inversoft-maintenance-mode-0.12.6.jar:0.12.6]
        at io.fusionauth.app.primeframework.FusionAuthAppPrimeServletContextListener.contextInitialized(FusionAuthAppPrimeServletContextListener.java:29) [fusionauth-app-1.3.1.jar:1.3.1]
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792) [catalina.jar:8.5.31]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256) [catalina.jar:8.5.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:8.5.31]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421) [catalina.jar:8.5.31]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411) [catalina.jar:8.5.31]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.lang.IllegalArgumentException: The configuration file [/usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/fusionauth.properties] doesn't exist.
        at com.inversoft.configuration.BasePropertiesFileInversoftConfiguration.initialize(BasePropertiesFileInversoftConfiguration.java:263) ~[inversoft-config-0.6.2.jar:0.6.2]
        at io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration.initialize(PropertiesFileFusionAuthConfiguration.java:88) ~[fusionauth-api-1.3.1.jar:1.3.1]
        at com.inversoft.configuration.BasePropertiesFileInversoftConfiguration.initializeAndLogErrors(BasePropertiesFileInversoftConfiguration.java:323) ~[inversoft-config-0.6.2.jar:0.6.2]
        at com.inversoft.configuration.BasePropertiesFileInversoftConfiguration.<init>(BasePropertiesFileInversoftConfiguration.java:64) ~[inversoft-config-0.6.2.jar:0.6.2]
        at io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration.<init>(PropertiesFileFusionAuthConfiguration.java:35) ~[fusionauth-api-1.3.1.jar:1.3.1]
        at io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration$$FastClassByGuice$$62935ad5.newInstance(<generated>) ~[fusionauth-api-1.3.1.jar:1.3.1]
        at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:82) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:125) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:80) ~[guice-4.2.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:173) ~[guice-4.2.0.jar:na]
        ... 16 common frames omitted
Jan 20, 2019 4:33:29 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class [io.fusionauth.app.primeframework.FusionAuthAppPrimeServletContextListener]
org.primeframework.mvc.PrimeException
        at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:77)
        at com.inversoft.maintenance.servlet.MaintenanceModePrimeServletContextListener.contextInitialized(MaintenanceModePrimeServletContextListener.java:39)
        at io.fusionauth.app.primeframework.FusionAuthAppPrimeServletContextListener.contextInitialized(FusionAuthAppPrimeServletContextListener.java:29)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

20-Jan-2019 16:33:29.046 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
20-Jan-2019 16:33:29.053 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
Jan 20, 2019 4:33:29 PM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class [io.fusionauth.app.primeframework.FusionAuthAppPrimeServletContextListener]
java.lang.NullPointerException
        at org.primeframework.mvc.guice.GuiceBootstrap.shutdown(GuiceBootstrap.java:88)
        at org.primeframework.mvc.servlet.PrimeServletContextListener.contextDestroyed(PrimeServletContextListener.java:47)
        at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4839)
        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5478)
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

20-Jan-2019 16:33:29.097 SEVERE [main] org.apache.catalina.core.StandardService.startInternal Failed to start connector [Connector[HTTP/1.1-auto-1]]
 org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-auto-1]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:440)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:793)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:681)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
Caused by: org.apache.catalina.LifecycleException: The connector cannot start since the specified port value of [-1] is invalid
        at org.apache.catalina.connector.Connector.startInternal(Connector.java:1011)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 11 more

20-Jan-2019 16:33:29.101 SEVERE [main] org.apache.catalina.core.StandardService.startInternal Failed to start connector [Connector[AJP/1.3-auto-3]]
 org.apache.catalina.LifecycleException: Failed to start component [Connector[AJP/1.3-auto-3]]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:440)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:793)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:681)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
Caused by: org.apache.catalina.LifecycleException: The connector cannot start since the specified port value of [-1] is invalid
        at org.apache.catalina.connector.Connector.startInternal(Connector.java:1011)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        ... 11 more

20-Jan-2019 16:33:29.103 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 5578 ms

[robotdan]

I only have on theory so far, the earlier log statements (shown below) confirm that when we start up tomcat we have not been able to read the config. In our setenv.sh we read the config file and set Java properties for the tomcat ports which are used in the server.xml file.

20-Jan-2019 16:33:21.285 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server}Setting property 'port' to '${fusionauth.management.port}' did not find a matching property.
20-Jan-2019 16:33:21.576 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'port' to '${fusionauth.http.port}' did not find a matching property.
20-Jan-2019 16:33:21.579 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'redirectPort' to '${fusionauth.https.port}' did not find a matching property.
20-Jan-2019 16:33:21.740 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'port' to '${fusionauth.https.port}' did not find a matching property.
20-Jan-2019 16:33:21.949 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'port' to '${fusionauth.ajp.port}' did not find a matching property.

Is it possible that the persistent volume is mounted after and thus hiding the directory that exists in the image?

One idea to narrow it down, can you mount the PV to a different directory other than /usr/local/fusionauth/config? And then see if it starts up and is able to find the config files?

Also, I see in your deployment you have waits in there for search and db. I theory this should not be necessary, we built all of that into our maintenance mode. FusionAuth will wait for a period of time to try to connect to the DB and search before we call it quits and fail to start up.

If you remove those wait checks and have any issues let us know, that should work.

[nadilas]

@robotdan I see your point. Mounting the pvc in a different directory does indeed let the server start up. Here's the new challenge 😄 I attached the log.
Issue 1: mysql connector cannot be downloaded and is therefore not present
Issue 2: the UI is not happy

Silent Configuration Mode Failed
Silent configuration of the database and search engine configuration was attempted and failed. Silent configuration was attempted because you provided the following environment variables:
DATABASE_URL
DATABASE_ROOT_USER
FUSIONAUTH_SEARCH_SERVERS
Possible reasons why this failed and you ended up here:
The database is not started, or not accessible using the JDBC URL provided in the DATABASE_URL environment variable.
The database root credentials provided by DATABASE_ROOT_USER and optionally DATABASE_ROOT_PASSWORD environment variables were incorrect or did not have adequate privileges to create the schema.
Elasticsearch is not started, or not accessible using the URL(s) provided in the FUSIONAUTH_SEARCH_SERVERS environment variable.
A problem exists between the keyboard and chair. (PEBKAC)
Please review the values you specified in the environment variables, ensure the database and Elasticsearch services are accessible and review the FusionAuth logs.

---

Regarding the waits, I was just being cautious. I didn't know if it would cause any problems, but depends_on is not a kubernetes concept, hence the workaround. Also the motivation was this note from your documentation:
Note that you MUST start the FusionAuth Search before starting the FusionAuth App service. You should also shutdown the Search Engine last. FusionAuth utilizes the Search Engine and database for persisting and retrieving users.
https://fusionauth.io/docs/v1/tech/tutorials/start-and-stop

---

Jan 20, 2019 7:43:41.914 PM ERROR com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - An attempt to download the MySQL connect jar failed. http://savant.inversoft.org/com/mysql/mysql-connector-java/8.0.12/mysql-connector-java-8.0.12.jar
java.io.FileNotFoundException: http://savant.inversoft.org/com/mysql/mysql-connector-java/8.0.12/mysql-connector-java-8.0.12.jar
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1890) ~[na:1.8.0_171]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[na:1.8.0_171]
	at java.net.URL.openStream(URL.java:1045) ~[na:1.8.0_171]
	at com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService.downloadMySQLConnector(JDBCMaintenanceModeDatabaseService.java:75) ~[inversoft-maintenance-mode-0.12.6.jar:0.12.6]
	at com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService$$FastClassByGuice$$a64dfb61.invoke(<generated>) [inversoft-maintenance-mode-0.12.6.jar:0.12.6]
	at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:51) [guice-4.2.0.jar:na]
	at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:85) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:125) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:80) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:173) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) [guice-4.2.0.jar:na]
	at com.google.inject.Guice.createInjector(Guice.java:87) [guice-4.2.0.jar:na]
	at com.google.inject.Guice.createInjector(Guice.java:69) [guice-4.2.0.jar:na]
	at com.google.inject.Guice.createInjector(Guice.java:59) [guice-4.2.0.jar:na]
	at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:58) [prime-mvc-1.11.1.jar:1.11.1]
	at com.inversoft.maintenance.servlet.MaintenanceModePrimeServletContextListener.contextInitialized(MaintenanceModePrimeServletContextListener.java:39) [inversoft-maintenance-mode-0.12.6.jar:0.12.6]
	at io.fusionauth.app.primeframework.FusionAuthAppPrimeServletContextListener.contextInitialized(FusionAuthAppPrimeServletContextListener.java:29) [fusionauth-app-1.3.1.jar:1.3.1]
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792) [catalina.jar:8.5.31]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256) [catalina.jar:8.5.31]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:8.5.31]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421) [catalina.jar:8.5.31]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411) [catalina.jar:8.5.31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Jan 20, 2019 7:43:42.329 PM INFO  com.inversoft.maintenance.DefaultMaintenanceModeSilentConfigurationWorkflow - 

---------------------------------------------------------------------------------------------------------
---------------------------------- Entering Silent Configuration Mode -----------------------------------
---------------------------------------------------------------------------------------------------------

Jan 20, 2019 7:43:42.527 PM ERROR com.inversoft.maintenance.db.DatabaseSilentConfigurationWorkflowTask - Protocol error.  Session setup failed.
Jan 20, 2019 7:43:42.529 PM INFO  org.primeframework.mvc.servlet.PrimeServletContextListener - Initializing Prime
Jan 20, 2019 7:43:42.682 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/fusionauth-app/apache-tomcat/../../config/fusionauth.properties]
Jan 20, 2019 7:43:42.683 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - Dynamically set property [fusionauth-app.public-url] set to [http://10.32.5.32:9011]
Jan 20, 2019 7:43:42.683 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - FusionAuth Runtime Mode [Production]
Jan 20, 2019 7:43:42.688 PM ERROR com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - An attempt to download the MySQL connect jar failed. http://savant.inversoft.org/com/mysql/mysql-connector-java/8.0.12/mysql-connector-java-8.0.12.jar
java.io.FileNotFoundException: http://savant.inversoft.org/com/mysql/mysql-connector-java/8.0.12/mysql-connector-java-8.0.12.jar
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1890) ~[na:1.8.0_171]
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[na:1.8.0_171]
	at java.net.URL.openStream(URL.java:1045) ~[na:1.8.0_171]
	at com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService.downloadMySQLConnector(JDBCMaintenanceModeDatabaseService.java:75) ~[inversoft-maintenance-mode-0.12.6.jar:0.12.6]
	at com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService$$FastClassByGuice$$a64dfb61.invoke(<generated>) [inversoft-maintenance-mode-0.12.6.jar:0.12.6]
	at com.google.inject.internal.SingleMethodInjector$1.invoke(SingleMethodInjector.java:51) [guice-4.2.0.jar:na]
	at com.google.inject.internal.SingleMethodInjector.inject(SingleMethodInjector.java:85) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:125) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:80) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:173) [guice-4.2.0.jar:na]
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) [guice-4.2.0.jar:na]
	at com.google.inject.Guice.createInjector(Guice.java:87) [guice-4.2.0.jar:na]
	at com.google.inject.Guice.createInjector(Guice.java:69) [guice-4.2.0.jar:na]
	at com.google.inject.Guice.createInjector(Guice.java:59) [guice-4.2.0.jar:na]
	at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:58) [prime-mvc-1.11.1.jar:1.11.1]
	at org.primeframework.mvc.servlet.PrimeServletContextListener.contextInitialized(PrimeServletContextListener.java:61) [prime-mvc-1.11.1.jar:1.11.1]
	at com.inversoft.maintenance.servlet.MaintenanceModePrimeServletContextListener.contextInitialized(MaintenanceModePrimeServletContextListener.java:45) [inversoft-maintenance-mode-0.12.6.jar:0.12.6]
	at io.fusionauth.app.primeframework.FusionAuthAppPrimeServletContextListener.contextInitialized(FusionAuthAppPrimeServletContextListener.java:29) [fusionauth-app-1.3.1.jar:1.3.1]
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792) [catalina.jar:8.5.31]
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256) [catalina.jar:8.5.31]
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:8.5.31]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421) [catalina.jar:8.5.31]
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411) [catalina.jar:8.5.31]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

[robotdan]

So I wonder if there is an additional paramter or configuration for the persistent volume to pickup the files that already exist, or perhaps you have to copy them I to the persistent volume. I'll have to do some additional reading on how kubernetes manages volumes.

Regarding the MySQL connector, that is strange. Does your instance have outbound network access? I am able to reach that URL. http://savant.inversoft.org/com/mysql/mysql-connector-java/8.0.12/mysql-connector-java-8.0.12.jar

However, i thought you were using PostgreSQL, in that case we should not be trying to download the MySQL connector jar.

The default jdbc string in the fuaionauth.properties file is for MySQL, so perhaps it is not picking up your database values from the environment variables and then falling back to the one in the config file.

[nadilas]

Thanks for trying to solve this!

The container should have web access, bit I’ll double-check that, maybe istio is blocking it.

However you are correct I have configured a postgres url, so there would be no need for the MySQL connector unless the config is the default one.

I wonder why the env variable is not getting picked up. To be honest it is a istio virtual service, but I can’t image why that would be interesting.

name: DATABASE_URL
value: jdbc:postgresql://fusionauth-db:5432/fusionauth

[nadilas]

hey @robotdan, so I did a few tests

1. Check the environment variables in the container -- OK

nadilas-MBP:fusionauth-infrastructure nadilas$ kubectl exec -it fusionauth-755689f548-gqmcw -c fusionauth -- bash
fusionauth@fusionauth-755689f548-gqmcw:/$ echo $DATABASE_URL
jdbc:postgresql://fusionauth-db/fusionauth
fusionauth@fusionauth-755689f548-gqmcw:/$ echo $DATABASE_ROOT_USER
postgres
fusionauth@fusionauth-755689f548-gqmcw:/$ echo $DATABASE_PASSWORD
hkaLBM3RVnyYeYeqE3WI1w2e4Avpy0Wd5O3s3

2. Install postgres client in container and access db directly -- turns out, the port name has to be anything but "http", so after changing that to "postgres" I can connect manually to the database -- OK

3. Check db user permissions by executing CREATE DATABASE fusionauth; -- OK

4. check search is available -- OK

fusionauth@fusionauth-84668fbc88-c5s4c:/$ curl -v http://fusionauth-search:9200
* Rebuilt URL to: http://fusionauth-search:9200/
*   Trying 10.35.255.73...
* TCP_NODELAY set
* Connected to fusionauth-search (xx.xx.xxx.xx) port 9200 (#0)
> GET / HTTP/1.1
> Host: fusionauth-search:9200
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 490
< server: envoy

5. Remove everything and redeploy FusionAuth from scratch

6. Start up
   Silent configuration failed

7. Only delete only -app pod and restart
   It seems like during startup in 6. the services are truly not accessible (probably not yet alive), I'll re-add the waiting initContainers in the afternoon --> TODO

But now the configuration is being seen, but cannot be parsed, which is weird?
Silent configuration failed:

Jan 22, 2019 7:23:23.007 AM ERROR com.inversoft.maintenance.db.JDBCURL - Could not parse jdbcString [jdbc:postgresql://fusionauth-db/fusionauth]
Jan 22, 2019 7:23:23.008 AM ERROR com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - Configuration [database.url] is invalid. It must begin with either jdbc:mysql: or jdbc:postgresql:

[voidmain]

Hi @nadilas. I believe that our JDBCURL class requires a port number in the URL. If you update your URL to this jdbc:postgresql://fusionauth-db:5432/fusionauth, it should work.

[nadilas]

Hi @voidmain, like magic! 😄

I updated the repo if you'd like to use it anywhere:
https://github.com/nadilas/fusionauth-k8s-infrastructure

The only thing remains, that the config will be generated each time, but I assume the database is not wiped, so it shouldn't be an issue. Maybe there's no need for the pvc for the config at all?

[robotdan]

That is great @nadilas! Did you end up deciding the wait-for stuff is necessary?

If you needed it to get it up and running I'd like to take a look at that again, I believe FusionAuth should be waiting for a bit to connect to the db and search.

However, perhaps this behaves different in k8s due to the networking config. FusionAuth will start up and enter maintenance mode if we cannot connect to the db or search. Once we enter that mode, we wait for the db and search ports to start accepting connections. This indicates to us that the services are up and running - we then attempt to connect to those ports and verify the db connection and search connections are healthy.

If K8s accepts connections on those ports as a proxy before the service is up and running - then perhaps are current methodology will not work and you'll need those wait statements. If that is the case, perhaps we want to enhance this feature to make it more resilient in a k8s environment.

Maybe there's no need for the pvc for the config at all?

I think you may be right, I'll review this a bit further to confirm. The one thing I know that is yet missing from being able to be set via an environment variable is the URL of each FusionAuth instance. In the fusionauth.properties file this is the fusionauth-app.public-url parameter.
https://fusionauth.io/docs/v1/tech/reference/configuration

We should be adding this in the near future so that it can be set in the docker-compose or k8s configuration. This value is used to let other nodes know when to reload a cache. We always fall back to a timed reload, so this won't break anything if not configured properly, but there may be a slight delay in all nodes updating a cache after an API key add or delete, or a JWT signing configuration change.

I updated the repo if you'd like to use it anywhere:

This is great. Is your example fairly generic such that we could just include in this repo? If so, do you want to submit a PR with all the code under a top level directory so it fits nicely into this repo? We could also link to your repo just to give folks an example to work from.

[nadilas]

@robotdan I can't be 100% sure that it's an availability issue, but without further debug log entries, that seems to be the case. I brought down the app once again and built it from scratch. Without the waiting the following error quits the silent configuration:

---------------------------------------------------------------------------------------------------------
---------------------------------- Entering Silent Configuration Mode -----------------------------------
---------------------------------------------------------------------------------------------------------

Jan 22, 2019 9:39:39.602 PM ERROR com.inversoft.maintenance.db.DatabaseSilentConfigurationWorkflowTask - The connection attempt failed.

So this issue remains. The root cause for this thread was me setting the postgres port name to http which made istio run off the rails, anything but http would have worked, it's now called postgres.

Whether the environment variables are picked up I cannot tell, since there's no indication for that in the log. It doesn't say which jdbc url was tried it just fails. A variable in the error message would be helpful to see if it's an env var/config or db availability issue, like: The connection attempt to ${url} failed.
With the waiting initContainers the silent mode configures accurately.

Also I removed the pvc completely on the fusionauth-app, it's simply not needed for my single instance so far. Being able to set the public url from an env var would be definitely appreciated for a multi-instance config. That being said, for kubernetes, the IP determined by the app should just do, (which in this case is a cluster-internal one) -- even though this is the one which gets external traffic routed into it:

Jan 22, 2019 9:51:33.915 PM INFO  io.fusionauth.api.configuration.PropertiesFileFusionAuthConfiguration - Dynamically set property [fusionauth-app.public-url] set to [http://10.xx.x.xx:9011]

The repo I posted above is fairly generic, it uses your docker-compose.yaml default password without a postgres root password and deploys to the default namespace. The only "extra" inside is the virtual service and gateway for istio. I can make a PR to have it in one-place (also a doc entry would be nice), but if you'd rather link it over I'm fine with that too, whichever you prefer.

[robotdan]

Thanks for the additional info @nadilas .

If you're up for it - a PR would be preferred. We can always modify it if necessary to be more generic, etc. If possible, also as part of your PR update the README with any necessary info for others to use to get it up and running.

When I get a chance I'll try to get a k8s cluster set up so I can try to replicate the issue when the wait-for options are removed. Perhaps we can enhance our maintenance mode startup to handle this internally.

A variable in the error message would be helpful to see if it's an env var/config or db availability issue, like: The connection attempt to ${url} failed.

We do log this in DEBUG, but the default config is ERROR so you won't see it in the log. We may want to add the ability to enable debug logging via environment variable to it is easier to debug this type of stuff when running in a container.

[nadilas]

Sounds good. PR is done