Reconsider nodes management

[ralphwetzel]

As of today, there are a number of node implementations that have to play together, to make node-red-mcu work. The following table tries to display what I am aware of, serving as a starting point to discuss how to organize this for the future into a sustainable concept.

#	implementation	required by	located @	purpose
1	sample-node.html	Node-RED	.node-red/node_modules/...	UI @ the node-RED editor
2	sample-node.js	Node-RED	.node-red/node_modules/...	implementation of the node's functionality for Node-RED runtime
3	class SampleNode	node-red-mcu	node-red-mcu/nodered.js	implementation of the node's functionality for node-red-mcu
4	mcu*sample-node	node-red-mcu-plugin	node-red-mcu-plugin/mcu-plugin.js	replacing sample-node.js in the Node-RED runtime to interface w/ node-red-mcu

[To be fair: This is a simplification - as there could be several node implementations within one sample-node.js file.]

Currently, there's an established concept for 1 & 2.
I'm proposing to discuss how to organize 3 & 4 (perhaps in combination w/ 1 and 2) and as well e.g. if to publish and install them as npm packages.
Consequentially this has influence on other aspects as well, e.g. the build environment setup.
Any ideas on how to proceed?

[phoddie]

Thanks for starting this conversation.

There are core (built-in) nodes and contributed (plug-in or external) nodes. This thread is about the contributed nodes only.

You list four components of a contributed node. There a few more:

5. Code to be used by nodered2mcu to optimize the configuration for the MCU. There's no standalone example of this as nodered2mcu doesn't support this capability yet.
6. The manifest.json used to build the MCU code
7. There's also the package.json used from npm

Let's use the classic lower-case example for discussion. Its directory contains:

• lower-case.html (1)
• lower-case.js (2)
• package.json (7)

I propose an mcu subdirectory to contain all files solely used with Node-RED MCU Edition. The presence of the mcu directory is an indication that this contributed node is compatible with the Node-RED MCU Edition. The mcu directory contains:

• manifest.json (6) to build the node. This includes any dependencies (for example, the MCU websocket node manifest includes the ECMA-419 WebSocket manifest)
• The source files and assets specifically for the MCU runtime (3). Note that there may be none if the module implementation for the full Node-RED (2) may also be used by the MCU.
• An optional JavaScript source file to transform the configuration in nodered2mcu (5). It may be enough for this to have a well known file name such as node2mcu.js.

As part of assembling the project's manifest.json, the tooling (your Node-RED MCU Builder plug-in) will update the includes section by adding the manifest for each plug-in node. This will optimize the code footprint of the device binary by including only those plug-in nodes used by the project.

[phoddie]

@ralphwetzel – the manifest does not include the file extension. mcconfig finds files that match (could be .js, .ts, .c, etc.). Your example above builds if the .js is excluded:

  "modules": {
    "*": [ "./example.es" ]
  }

The default module specifier is derived from the filename with the file extension removed. Hence, here the module is importable using the module specifier "example.es":

import example from "example.es"

To use the module specifier "example" instead, remap in the manifest:

  "modules": {
    "example": "./example.es"
  }

import example from "example"

Regarding the issue you report with the *.map file, I'm unsure how to reproduce that. I'd be happy to take if a look if you would share a test case. Thank you.

[ralphwetzel]

@phoddie: I'm changing the way to create the build directory. Whereas I used to copy "everything" from node-red-mcu into that (temporary) directory, I intend to "just" generate a manifest.json and use this to reference to the one defined by node-red-mcu. That works perfect - despite one aspect:

The given manifest.json references a flows.json in the same directory:

node-red-mcu/manifest.json

Lines 19 to 28 in 01470d8

	"modules": {
	"*": [
	"./main",
	"./nodered",
	{
	"source": "./flows",
	"transform": "nodered2mcu"
	}
	]
	},

I yet intend to operate with different flows.json file (located in the created build directory). As consequence, I end up with two flow.jsons that are processed by nodered2mcu ... and the first (empty) file wins (currently).

I tried to opt-out the flows.json of the node-red-mcu directory using the "~" option ... without success:

"modules": {
    "*": [
      {
        "source": "./flows",
        "transform": "nodered2mcu"
      }
    ],
    "~": "$(MCUROOT)/flows"
  }

Did I fail to understand the idea behind "~" ... or do you spot another error?

[phoddie]

The ~ feature of manifests is always a little tricky. Here I don't think we should use it. Rather, we should have a manifest specifically for the Node-RED MCU runtime and a separate application manifest that includes it. That makes your work simpler. I committed a first pass at that. the manifest_runtime.json file does not include main or flows so you can organize those however you like.

[ralphwetzel]

The ~ feature of manifests is always a little tricky

It is - and I made it (finally) working. In this case, the reference to the transform property is necessary:

"~": [
      {
        "source": "$(MCUROOT)/flows",
        "transform": "nodered2mcu"
      }

Anyway. The setup becomes fairly cleaner when splitting the manifests. Thank you for that step!

[phoddie]

Glad that worked.

[ralphwetzel]

@phoddie:
I'm still working through the jungle of auto-building manifests 😄, currently sorting out some details:

When including a module that has a .ts file next to a .js in the same directory, mcconfig is generating two (2) make commands, the second one to compile the .ts file to .js:

$(MODULES_DIR)/[...]/logger.xsb: [...]/logger/src/index.js
	@echo "# xsc [...]/logger.xsb"
	$(XSC) [...]/logger/src/index.js -d -c -e -o $(@D) -r logger

$(MODULES_DIR)/[...]/logger.xsb: $(MODULES_DIR)/index.js
	@echo "# xsc [...]/logger.xsb"
	$(XSC) $(MODULES_DIR)/index.js -d -c -e -o $(@D) -r logger
%/index.js : %/tsconfig.json 
	@echo "# tsc tsconfig.json"
	tsc -p $(MODULES_DIR)/tsconfig.json

This triggers two warnings

[...]/makefile:1019: warning: overriding commands for target `[...]/logger.xsb'
[...]/makefile:787: warning: ignoring old commands for target `[...]/logger.xsb'

and then fails w/ make: tsc: No such file or directory ... as there's no path to tsc given.

In this case, the tsc action is not necessary - as the .js file is already in place.

Is there a flag to tell mcconfig not to care for .ts files ... preferably definable per manifest.json?

[phoddie]

@ralphwetzel – Thanks for the report from the jungle. You have bumped into an interesting problem. As you know, the manifest doesn't contain file extensions and mcconfig figures out what to do based on the file matches it finds. This is often convenient -- a module with both JavaScript and C implementations is included as "./module" and both "module.c" and "module.js" are built. We didn't anticipate a situation where there would be two different implementations of the same code (e.g. "./module.ts" and "module.js"). In this case, I'm assuming you are working with a package that provides both the TypeScript source and the transpiled JavaScript version of that. Some options when both are present:

• Always use the JavaScript version. This will work even if TypeScript is not installed.
• Always use the TypeScript version. This makes the type information available to other TypeScript sources in the project.
• Fail, since the situation is ambiguous. This is very safe but won't work with some packages.
• Provide a way in the manifest to specify which to use -- this could be required when there is a conflict or could be optional with some default (e.g. use the TypeScript version)

I'm not sure.... what do you think?

[ralphwetzel]

Indeed, your current strategy is very convenient & powerful seen from the developer's perspective. As this is a very edge case, I propose to implement a way to opt-out the transpilation of the TypeScript source - when & after the above mentioned error was hit. I've seen there's already a typescript property defined for manifest.json. Based on this, a flag like

typescript: { "compiler": "false" }

would fully meet my (current) demands.

There could be alternative implementations as well:

• Check if the potential result of the tsc operation is already present (in the same directory) & then skip the transpilation.
• Provide a way to opt-out files from being processed at all.

The first one fits into the idea of

mcconfig figures out what to do based on the file matches it finds

whereas I'd prefer to have a bit more control over what's happening in that case.
The second one has further potential beyond this issue, yet requires some more effort for implementation.

[ralphwetzel]

I've pushed the next iteration of the plugin into the repo. There's as well a bit of documentation that discusses what the plugin does regarding the manifest creation process. Just be warned: The number of loose ends & incomplete implementations is significant.

[ralphwetzel]

• The proxy isn't working yet, but the additional logging may offer some clues for you:

- Connecting to xsbug: Resolved localhost to ::1/6.

That confirms my assumption. Should be fixed soon...

[ralphwetzel]

Should be fixed soon...

Is fixed now.

[phoddie]

It is! So great! I used Build Target "All". With that, it deploys, xsbug works, inject works from the Node-RED Editor, and debug messages are sent to the Node-RED Editor. Thank you!

[phoddie]

There is an interesting second-order bug. After running a flow on the MCU, I can no longer run that flow in Node-RED. When I try, it deploys but no debug output appears and inject seems to be ignored. If I manually remove the _mcu properties from the nodes, then it works again with Node-RED.

[ralphwetzel]

There is an interesting second-order bug. After running a flow on the MCU, I can no longer run that flow in Node-RED.

Do you still have this issue with the latest changes of the plugin?