Credentials in configuration nodes

[mtoko]

I am using an MQTT node with credentials and I vaguely remember that the configuration node credentials are only sent on the first deploy, so before the node mcu plugin the procedure was to copy the flow into node red mcu before deploy so that these were downloaded with the flow to the mcu. I assume now that I have the plugin working that the config node credentials would be downloaded.
Summary
I have a simple flow of an inject node and an MQTT out node to a MQTT broker that is known to work. The flow works correctly if there are no credentials in the MQTT out or the broker but does not connect if the node red mcu flow has credentials. I have tried to input credentials and then build before deploy (savings config setup) and also to build after deploy but neither works
The mcu is a ESP32/nodemcu

Example Flows
This is the flow before deploy, you can see the credentials in the broker
[ { "id": "9c629d06eeea7a09", "type": "tab", "label": "Flow 1", "disabled": false, "info": "", "env": [], "_mcu": { "mcu": true } }, { "id": "e42e258e6f3d5c8c", "type": "inject", "z": "9c629d06eeea7a09", "name": "", "props": [ { "p": "payload" }, { "p": "topic", "vt": "str" } ], "repeat": "20", "crontab": "", "once": false, "onceDelay": "5", "topic": "", "payload": "", "payloadType": "date", "_mcu": { "mcu": true }, "x": 170, "y": 80, "wires": [ [ "da80f24bceaa4e51", "2b5ec21c66d9da1b" ] ] }, { "id": "da80f24bceaa4e51", "type": "mqtt out", "z": "9c629d06eeea7a09", "name": "", "topic": "Testy76", "qos": "0", "retain": "false", "respTopic": "", "contentType": "", "userProps": "", "correl": "", "expiry": "", "broker": "9a98af129a0f5b68", "_mcu": { "mcu": true }, "x": 520, "y": 40, "wires": [] }, { "id": "2b5ec21c66d9da1b", "type": "debug", "z": "9c629d06eeea7a09", "name": "debug 1", "active": false, "tosidebar": true, "console": false, "tostatus": false, "complete": "false", "statusVal": "", "statusType": "auto", "_mcu": { "mcu": true }, "x": 500, "y": 160, "wires": [] }, { "id": "9a98af129a0f5b68", "type": "mqtt-broker", "name": "", "broker": "192.168.1.107", "port": "1883", "clientid": "", "autoConnect": true, "usetls": false, "protocolVersion": "4", "keepalive": "60", "cleansession": true, "birthTopic": "", "birthQos": "0", "birthPayload": "", "birthMsg": {}, "closeTopic": "", "closeQos": "0", "closePayload": "", "closeMsg": {}, "willTopic": "", "willQos": "0", "willPayload": "", "willMsg": {}, "userProps": "", "sessionExpiry": "", "_mcu": { "mcu": false }, "credentials": { "user": "user", "password": "password" } } ]
And this is the flow after deploy, no credentials
[ { "id": "9c629d06eeea7a09", "type": "tab", "label": "Flow 1", "disabled": false, "info": "", "env": [], "_mcu": { "mcu": true } }, { "id": "e42e258e6f3d5c8c", "type": "inject", "z": "9c629d06eeea7a09", "name": "", "props": [ { "p": "payload" }, { "p": "topic", "vt": "str" } ], "repeat": "20", "crontab": "", "once": false, "onceDelay": "5", "topic": "", "payload": "", "payloadType": "date", "_mcu": { "mcu": true }, "x": 170, "y": 80, "wires": [ [ "da80f24bceaa4e51", "2b5ec21c66d9da1b" ] ] }, { "id": "da80f24bceaa4e51", "type": "mqtt out", "z": "9c629d06eeea7a09", "name": "", "topic": "Testy76", "qos": "0", "retain": "false", "respTopic": "", "contentType": "", "userProps": "", "correl": "", "expiry": "", "broker": "9a98af129a0f5b68", "_mcu": { "mcu": true }, "x": 520, "y": 40, "wires": [] }, { "id": "2b5ec21c66d9da1b", "type": "debug", "z": "9c629d06eeea7a09", "name": "debug 1", "active": false, "tosidebar": true, "console": false, "tostatus": false, "complete": "false", "statusVal": "", "statusType": "auto", "_mcu": { "mcu": true }, "x": 500, "y": 160, "wires": [] }, { "id": "9a98af129a0f5b68", "type": "mqtt-broker", "name": "", "broker": "192.168.1.107", "port": "1883", "clientid": "", "autoConnect": true, "usetls": false, "protocolVersion": "4", "keepalive": "60", "cleansession": true, "birthTopic": "", "birthQos": "0", "birthPayload": "", "birthMsg": {}, "closeTopic": "", "closeQos": "0", "closePayload": "", "closeMsg": {}, "willTopic": "", "willQos": "0", "willPayload": "", "willMsg": {}, "userProps": "", "sessionExpiry": "", "_mcu": { "mcu": false } } ]

[ralphwetzel]

@phoddie:

After @mtoko raised this topic, I began considering options to provide the credentials for further processing. So far, I see three potential ways to do this:

1. Add the credentials to the configuration of the node in flow.json.
2. Create an additional file (credentials.json) that provides the credentials. The credentials could be stored in an object, with <node>.id as key.
3. Put the credentials into manifest.json.

I don't like the first option. There's a (not so small) risk that people will post this data somewhere without removing the credentials first.

Option 2 is nice. We even could encrypt the credentials. The encryption key could be stored in the configuration of the node in flows.json. Yet - despite this data could as well be created without using the plugin - it wouldn't be really "user friendly".

Option 3 looks like the most promising for me:

• There's no need to manage another file.
• There're most probably as well other credentials already in manifest.json.
• There's an established way to get the data from within a flow.
• Users could easily edit manifest.json to add the credentials.

The credentials could be stored within "config", again with <node>.id as key, e.g.:

"config": {
	"noderedmcu": {},
	"sntp": "pool.ntp.org",
	"9a98af129a0f5b68": {
		"user": "user",
		"password": "password"
	}
},

What do you think about this proposal?
Do you see another option to provide these credentials?

[phoddie]

@ralphwetzel – thank you for the investigation and implementation work. I completely agree that we should try to address this challenge. Your proof-of-concept shows that a solution is in reach. As you describe, the challenge is the form that should take.

My initial impression is that I'd rather not modify the structure of the flows.json exported by Node-RED. Doing so creates the potential for conflicts and incompatibilities in the future.

There are other credentials that will need to be exported – TLS configurations and HTTP requests with authentication enabled, for example. I'm unsure how those will work. I think that's worth looking into, so there's the potential for a consistent solution.

There is a risk with the proposal to store the credentials in config section of the manifest. That data is effectively a global variable at runtime, accessible to any script – including Function nodes – by importing the mc/config module. An optimal solution would limit runtime visibility of the credentials to only those nodes & modules which need them.

I don't have a suggestion at the moment. I'll have to give the some more thought.

[ralphwetzel]

Hi @phoddie !

There are other credentials that will need to be exported

Scanning the node-red core, credentials are held by tls-config, http proxy, mqtt-broker & http request. There are a few more nodes managing credentials in node-red-nodes. All of them use the "official" credentials interface. As long as this interface is used, the logic already prepared is able to get the credentials and make them available.

An optimal solution would limit runtime visibility of the credentials to only those nodes & modules which need them.

Very fair point!

There's another one we need to consider as well: There might be a contrib node (that node-red-mcu wraps into a Compatibility node) that handles as well credentials! A contrib node - most probably - expects to access his credentials via this.credentials.[...]. Consequentially, we (would) need to keep this interface - or at least support it in the Compatibility node.

Trying to connect those thoughts:

• Putting the credentials into flows.json creates the risk of unintended disclosure.
• Making them available in config exposes them to all nodes in the flow.
• this.credentials.[...] needs to be supported - at least for the Compatibility node.
• What if ... credentials are provided as additional parameter in the modules section of manifest.json?

{
    "source": "./flows",
    "transform": "nodered2mcu"
    "credentials" : {
        "9a98af129a0f5b68": {
            "user": "user",
            "password": "password"
        }
    }
}

This would allow to pass the credentials to nodered2mcu, like ...

    else if (parts.extension == ".json") {
        if ("nodered2mcu" === query.transform)
            this.appendFile(tool.nodered2mcuFiles, target, source, include, query.credentials);
    }

... and nodered2mcu would be able to put the credentials into the config of the dedicated node?

[phoddie]

Thanks for looking into this further. The credentials mechanism does seem to be how Node-RED manages this. The MCU runtime already expects that the credentials property is present when the node is instantiated. So, the question is how to manage that.

Using the manifest to relay the credentials is a reasonable option. It would put more knowledge of Node-RED into mcconfig and mcrun. I would prefer to avoid expanding the scope of those tools because they are foundational and already quite complex. More of a concern, perhaps, is that just as developers share their flows.json files, they also share their manifest.json. So, maybe it is worth considering approaches which segregate the sensitive information to a unique file to further minimize the risk of a mistake being bad.

Node-RED itself separates the flow and JSON into flows.json and flows_cred.json. The runtime knows how to discover and merge the credentials file into the flow We could do the same, for example by having aflows_cred_mcu.json file placed next to flows.json. It would be auto-discovered by nodered2mcu and merged into the generated JavaScript. The contents could be along the lines of your credentials property above:

{
    "credentials": {
        "9a98af129a0f5b68": {
            "user": "user",
            "password": "password"
        }
    }
}

[ralphwetzel]

We could do the same, for example by having a flows_cred_mcu.json file placed next to flows.json.

Let's do it that way.

Would you agree to name this file "credentials.json" - to explicitly indicate its contents?
As this file should hold nothing but the credentials, I'd simplify the data structure a bit:

{
    "9a98af129a0f5b68": {
        "user": "user",
        "password": "password"
    },
    "aaaabbbbccccdddd": {
        "user": "user",
        "password": "password"
    }
}

I'll create an initial implementation based on our exchange. Don't hesitate to raise your concerns if you have any regarding those two last details...

[ralphwetzel]

I'll create an initial implementation

Done. It's in the repo now...
@phoddie: Your feedback is appreciated!

[phoddie]

I'd prefer to have a root "credentials" element in the JSON, as that will offer a degree of future-proofing by allowing additional information to be added, version number, etc. The size isn't a concern here.

I'm a little surprised you are proposing "credentials.json" over "flows_cred_mcu.json" I recall you favoring less generic naming. ;) Having the name begin with "flows" ensure it will appear in the directly adjacent (or nearly so) which helps reinforce that they go together. I thought starting with "flows_cred" which matches the Node-RED file with the same purpose would aid people in understanding what the file does.

[ralphwetzel]

I'm a little surprised you are proposing "credentials.json" over "flows_cred_mcu.json" I recall you favoring less generic naming. ;)

😄 I thought, being explicit has a benefit here ... despite it might be generic.

I'd prefer to have a root "credentials" element in the JSON

No problem.

I'll commit a change in a few minutes...

[colinl]

Sorry, late to the party here. Am I misunderstanding something or is the proposal that the user must add a separate credentials file for node-red-mcu to use? Cannot we use the existing flows_cred.json and provide just the secret required to decrypt it. In fact the plugin can pick that up automatically.

[ralphwetzel]

Cannot we use the existing flows_cred.json

This depends on what you understand as "use the existing ...".

The current idea is to create a separate file with just the credentials applicable for the node(s) to be forwarded to node-red-mcu. We need these (decrypted) to include them into the mcu build. I'd vote against exposing more - and especially against exposing a Node-RED (system managed) secret.

The plugin is going to create this file. If of interest for you: Just pull the latest commit of the plugin. It's already creating this file - if any node holds credentials.

[phoddie]

@ralphwetzel – I had time to give the updated plug-in a try this evening. The flows_cred_mcu.json output matches what I expected. Nice work. It should be straightforward from here. I'll try to get to that in the coming days.

[phoddie]

@colinl – in response to your comments, I would describe the approach we are taking as using the existing flows_cred.json by transforming them into a format that can be used by the Node-RED MCU Edition tooling (specifically, nodered2mcu). For users of the plug-in, @ralphwetzel has made that entirely transparent.

[phoddie]

@ralphwetzel – One last thought. Since the flows_cred_mcu.json document is not intended to be shared, the plug-in could place a warning at the top. Since this is JSON, it cannot be a comment, but something like:

{
    "warning": "DO NOT SHARE THIS FILE. It contains passwords and other sensitive data",
    "credentials": {
    }
}

[colinl]

That is what I don't like about this solution, the credentials being saved in plain text. Is the file regenerated at each build? Could it be deleted after the build is complete? If you know when the build is complete that is, perhaps you don't know that.

[phoddie]

the credentials being saved in plain text

I see. That wasn't clear to me before.

I think this is the best we can do at the moment. The file is set-up with enough flexibility that it can evolve in a compatibility way in the future.

Could it be deleted after the build is complete?

The plug-in could do that (up to @ralphwetzel). At that point, the credentials will still exist in plaintext in the generated flows.js and in the generated binary. So, deleting that file will lower the exposure of the information, but not eliminate it.