architect plugins / macro hooks brainstorm

[filmaj]

Is your feature request related to a problem? Please describe.
I would like for macros to have the ability to do more than augment the cloudformation JSON. For example, for macros that add new Lambda functions, it would be nice to hook into @architect/create so that the lambda function directories can be generated if missing.

Describe the solution you'd like
Macros currently allow into @architect/deploy's lifecycle - after @architect/package has run to generate a baseline CloudFormation JSON template, but before the deploy is executed. In this manner, macros can "augment" the CloudFormation template pre-deploy.

Given this macro baseline, it feels natural to extend the ability for macros to do other things. Some ideas on what kind of lifecycle events / hooks for macros would be cool, with some initial implementation notes:

• @architect/create and hydrate. Have arc create (and hydrate) lambda function directories for the macros that add new lambdas to your project.
• @architect/sandbox. Support for local dev for your macros!
• @architect.logs. Pull logs down from the cloud for the macros that interact with other AWS services.

How could this look like? Currently a macro is a node module that exports a function with a particular parameter signature:

module.exports = async function mymacro (arc, sam, stage='staging') {
  // modify sam cloudformation here 
  console.log({ arc, sam, stage })
  return sam
}

Perhaps this function could be augmented with specific properties, themselves functions, that then get looked for in the various @architect packages. Some implementation ideas for the specific use cases covered above:

• @architect/create. The main create function works by:
  1. retrieving an @architect/inventory instance and looking at the core supported Lambda-backed arc pragmas
  2. creates any directories for the lambda-backed pragmas (using an implied src parameter provided by inventory to figure out the path to said directory)
  3. writes out a stub handler file using templates hard coded into @architect/create.
  • One idea on how to implement create support for macros:
    • Also look up the macros from the inventory (alongside the core supported pragmas) and require() them.
    • Have a macro additional function, which gets passed an instance of inventory and perhaps which exists within the macro at module.exports.create, that returns an array of objects. Each object represents a lambda-backed directory that the macro needs within the arc project. These objects would contain:
      • a src property (pointing to the directory of the lambda code. Using the IoT Rules macro i've recently created, this would look like src/rules/name-of-rule)
      • a body property containing the template of the lambda handler file
    • Finally, create could use this macro output and leverage the existing src/lambda code (with minor adjustments) to create default handler directories and files for macros.
• @architect/hydrate. Not much would need to be changed here as hydrate's implementation is pragma-agnostic. The only hiccup would be that, currently, any lambda directories not listed in inventory.lambdaSrcDirs get ignored and not hydrated - so this filter either needs to be augmented to allow for macro directories to also not be ignored (by e.g. using the above create implementation suggestion, hydrate could invoke the macro's create hook to determine which additional source directories the macro registered that may also need hydration (i.e. require('./src/macros/macro-iot-rules').create(inventory) would return e.g. [ { src: 'src/rules/device-connections', body: 'module.exports = function() {}' } ], which could then be mapped to an array of macro-specific lambda source directories via map(o => o.src)), or perhaps by allowing macros to modify the inventory (more on that below).
• @architect/logs. The get Logical Id module inside logs hardcodes two things:
  1. whether to read logs from a path based on whether inventory has the path registered inside lambdaSrcDirs (so same limitation as hydrate)
  2. which pragmas to read logs from (so same limitation as create)
  • ... both of these are solvable problems, as discussed above.
• @architect/sandbox. This is the doozy! Local dev support for macros is the holy grail to chase in this issue. Looking at the pattern for how http, events and tables are supported in sandbox, it looks like there is a start() and end() method for each of these services. Perhaps we could use this same approach for macros too? On sandbox startup, macros could be required in, and if they contain a start function handler, that would then be executed. We could do the same thing on sandbox shutdown, pull the macros in, and if they contain an end function handler, then that gets executed.
• @architect/inventory. Perhaps it is worth providing macros a hook into inventory as well? In hydrate and logs, inv.lambdaSrcDirs is used to filter out which locations to operate on. Thus, perhaps there is benefit in allowing macros to modify lambdaSrcDirs to 'register' their own lambda source directories (if applicable to the macro) - and thus get support for macro hooks into both hydrate and logs 'for free' in that sense?
• Tests for all the above package changes.
• Documentation for plugin authors.

There are probably also arguments to be had to support hooks for other arc packages (deploy and destroy), but I lack specific use cases for these for now so I will leave them out.

FYI, I intend to prototype support for these in forks in the coming days / weeks.

Describe alternatives you've considered
This issue is a starting point, but certainly would like to collect more thoughts on this!

Additional context or notes
Started as a conversation in Slack: https://architecture-as-text.slack.com/archives/CMK98634P/p1610647437008800

[filmaj]

Update: I have a working prototype of at least the create integration worked out:

• The plugins branch of our IoT Rules macro has an additional exported method that, based on the arc app inventory, instructs @architect/create which directories to create and how to populate them.
• The plugins branch of our create fork will pull in macros, execute any create handlers the plugin/macro has, and feed the output of that into the lambda code generator (see GitHub's compare view for full details).

[filmaj]

Update part deux: hydrate is working using a fork of inventory that adds the macro-registered lambda src directories into the lambdaSrcDirs array that inventory provides. Full compare view for inventory here. Nice thing about this is that it necessitated no changes to hydrate other than the updated inventory package.

[filmaj]

Got logs working (compare of logs here), though so far following the implementation I've put together in this prototype, this elucidates two requirements of macro/plugin authors:

1. a requirement in the naming scheme of the CloudFormation JSON: you have to name your macro lambda function resources <Whatever>MacroLambda, i.e.

    "TestMacroLambda": {
      "Type": "AWS::Serverless::Function",
      "Properties": { ... }
   

2. you must also provide a name property along with src and body as the output lambda objects for the create handler. name will get mapped to the logical resource name, src points to the directory containing the lambda code and body is the template for the lambda handler. Reference implementation of this kind of macro/plugin available in the plugin I'm working on: macro-iot-rules.

So rough conclusion here is that there will be some conventions that plugin authors will need to follow that will need to be documented.

[filmaj]

Got the hooks in for sandbox support, but now thinking it might be nice to also expose a key handler to a macro's sandbox service.

Perhaps something like:

1. Start sandbox
2. Macro service's start handler runs (e.g. kicking up some kind of server in the background)
3. If user presses the M key inside sandbox, it renders a list of macro names that export a sandbox key handler (perhaps this list is shown to the user via something like the enquirer npm package)
4. If user selects a specific macro from the list, the macro's key handler gets invoked (perhaps being passed an instance of enquirer in case the macro wants to further prompt the user); this allows the macro to simulate events or do other local dev-y things to aid in development.

Would love any feedback / comments / suggestions on this!

[filmaj]

Thinking through this a bit more, with macros being given sandbox start and end hooks, that should be sufficient for macros to do whatever they want in terms of local testing in sandbox. While sandbox could expose some formal interface to macros (for e.g. attaching key handlers or hooking into some more generic sandbox macro selector process as per my last comment), a macro's sandbox start handler could do this on its own (like attach to process.stdin).

So perhaps for now sandbox start and end is sufficient? If so, after I add tests and docs, I am ready to send PRs for this...

One thing that probably needs refactoring is the ability to require macros in - right now this code (originally from the deploy package) is copy-pasted all over the place in the forks I'm working in for this support. Perhaps worth factoring out somewhere? I'm thinking either utils, or perhaps as a core part of inventory's intermediary format, could work. Thoughts?

[filmaj]

Additional idea came in via Thomas on Slack was to expose sandbox's http server to macros, so that macros can hook up middleware to it if they want (use case provided as an example was a cognito authorizer). Noted!

[filmaj]

All of these ideas have been integrated into a more formal and fleshed out RFC which is over at #1062. Closing this in favour of the new issue.