Feature Request: Easier Ability to use Environment Variables in Configuration

[tillig]

I've been using Magic Mirror 2.8.0 and want to store secrets (e.g., OpenWeather API keys) outside of my configuration file so I can commit the file to source control. This would also allow for easier Docker image creation where you could poke in some environment variables rather than full config files.

Doing this has been a challenge.

• The config.js is used in both a server and a client context. That is, instead of the Electron client app retrieving configuration from the /config endpoint of the server, the config.js is used directly. This means any references to process.env don't work because...
• Electron doesn't provide the process environment on the server to the running client. [This issue shows an example of that.] process doesn't exist by default, and even if you enable nodeIntegration, the process that Electron is running under isn't the same as the process the server started under. Instead, you need to require('electron') and access the electron.remote.process.env to get those variables marshaled over. But...
• Electron nodeIntegration is off by default. It's also turned off in the main electron.js. This can be overridden by config.

There is a discussion where I raised this over in the forum.

A working config.js with environment variables looks like this:

const owApiKeyName = "OPENWEATHER_API_KEY";
var remote = null;
if (typeof window !== "undefined") {
  remote = window.require("electron").remote;
}

var readEnvironment = function (name) {
  if (typeof process !== "undefined" && process.env[name]) {
    // process is undefined in the Electron app.
    return process.env[name];
  }
  if (remote && remote.process.env[name]) {
    // remote is null if the Electron nodeIntegration value isn't set to true.
    return remote.process.env[name];
  }
};


var config = {
  address: "localhost",
  port: 8080,
  ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"],
  language: "en",
  timeFormat: 24,
  units: "imperial",

  electronOptions: {
    webPreferences: {
      nodeIntegration: true
    }
  },

  modules: [
    {
      module: "alert"
    },
    {
      module: "updatenotification",
      position: "top_bar"
    },
    {
      module: "clock",
      position: "top_left"
    },
    {
      module: "calendar",
      header: "US Holidays",
      position: "top_left",
      config: {
        calendars: [{
          symbol: "calendar-check",
          url: "webcal://www.calendarlabs.com/ical-calendar/ics/76/US_Holidays.ics"
        }]
      }
    },
    {
      module: "compliments",
      position: "lower_third"
    },
    {
      module: "currentweather",
      position: "top_right",
      config: {
        location: "Hillsboro",
        locationID: "5731371", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city
        appid: readEnvironment(owApiKeyName)
      }
    },
    {
      module: "weatherforecast",
      position: "top_right",
      header: "Weather Forecast",
      config: {
        location: "Hillsboro",
        locationID: "5731371", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city
        appid: readEnvironment(owApiKeyName)
      }
    },
    {
      module: "newsfeed",
      position: "bottom_bar",
      config: {
        feeds: [{
          title: "New York Times",
          url: "http://www.nytimes.com/services/xml/rss/nyt/HomePage.xml"
        }],
        showSourceTitle: true,
        showPublishDate: true,
        broadcastNewsFeeds: true,
        broadcastNewsUpdates: true
      }
    }
  ]
};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") {
  module.exports = config;
}

The idea is that you have to fall back from process to remote.process before you can get to the environment.

I'd like to propose a couple things that would make environment variable usage easier:

• Separate client and server configuration such that...
  • The client only needs the endpoint to query and the Electron options.
  • The server has the endpoint to serve on and the module configuration.
• Change the Electron app to query module configurations from the server rather than directly using config.js.

From a backwards compatibility perspective, you could treat config.js as something that both use and the difference would be that the module settings on the Electron side that are read from config.js would be ignored in favor of retrieved settings.

[tillig]

I'm finding that the nodeIntegration: true bit really hoses up modules. The default clock module, for example, stops loading things that have time zones set because the node_modules path for the server isn't found by the client. Or something. I'll have to find a different workaround.

Regardless, having environment variable support would be huge. I think the client getting the configuration from the server /config endpoint would address a lot.

[tillig]

I found that the clientonly code does try to retrieve configuration from the server but only if the server isn't local. If I remove the constraint for a non-local server (so it always reads remote config), I can see that the configuration gets read there and does have the environment variables from the server... but once it hits the modules, that configuration isn't actually used.

I added some logging to look at configuration and it seems all the way through electron.js the values are correct, but when the modules are running, they somehow have the wrong config - config they've read from config.js rather than the /config remote endpoint.

As a side note, when running in clientonly mode it doesn't appear there's a way to get dev tools running. I'll file a separate issue for that.

[tillig]

I think the problem is server.js - instead of including the JSON from the /config endpoint, it replaces a #CONFIG_FILE# token with the config.js location and assumes it can just be read directly.

This bypasses all the environment and loads config.js directly.

You can fix that in a kind of hacky way by creating a "module" in a different location using the configuration in server.js:

  app.get("/config-module/config.js", function (req, res) {
    var moduleConfig = "var config = " + JSON.stringify(config) + ";";
    moduleConfig += "if (typeof module !== \"undefined\") { module.exports = config; }";
    res.type('text/javascript').send(moduleConfig);
  });

Then in app.get("/"...) where the HTML is replaced with the location of config.js, use that generated module instead:

html = html.replace("#CONFIG_FILE#", "/config-module/config.js");

Or, better, just put the generated module in index.html directly and don't read the config.js at all.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tillig]

It appears this is still something that would be helpful.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[buxxi]

Would also like this functionality for the same reason as @tillig
Haven't really looked into how the core of the MagicMirror works but if it just includes the config-file in both the client and server it feels kind of wrong.

Could the notification-system be used for this where a LOAD_MODULE-notification is sent on startup that triggers the start()-method on that module. Then the payload could be that modules configuration only with environment-variables populated.

And wouldn't it be better with having some kind of templating instead of functions in the config, like:
config: { appid: "${env.OPENWEATHER_API_KEY}" }