RFC: Lambda macros

[djgrant]

context

lambda modules compile to two targets – an infrastructure definition and the actual runtime module.

to achieve this the compiler needs to create two modules – one that can be evaluated during the creation of the orchestration graph, and another that is only ever evaluated at runtime.

challenges

currently, the compiler just looks for the config export, and statically parses its values. to be able to infer more about the desired infrastructure, more capabilities are required. this has been discussed in #2.

this RFC explores the idea of using macros to create safe code paths that are capable of building more capable features.

approach

generally, a macro is a function that can be evaluated at compile time, modifying the resulting compiled code.

as lambda modules have two compile paths, the lambda macros will also define two implementations – one implementation that modifies the compiled infrastructure module, and another that modifies the compiled runtime module.

spec

• functions with different implementations for infra and runtime compile paths
• macros are identified by their import path i.e. @notation/**.runtime & $ prefix
• macros may only be used at the top level of .fn files
  • ensures that their impact is visible e.g. imported packages are not able to discreetly change authorisation setup
• infra implementations may:
  • modify the compiled infra lambda module, particularly the config objects
• runtime implementations may:
  • replace the function arguments at compile time
  • add import statements

questions

part of what will makes lambda macros useful, is being able to reference other infrastructure resources within them (and have access to their outputs).

this raises the question of whether those imported infrastructure resources should be referenced by runtime code that is also has the import in scope.

possible approaches:

• silently remove references (no, going to unpredictably break user code)
• runtime error (no, a compile-time error is preferable to a runtime error)
• compile error (maybe, if there's also a linter error, although counter intuitive)
• allowable at runtime (maybe, but is this safe?)

what would it mean to have infra resources available at runtime?

• risks exposing sensitive data
• could provide a more direct way to pass useful values into modules rather than via KMS – but is it safe?
• is there a way to sanitise resources so that at runtime, the object only exposes outputs when explicitly requested?

proposal

example usage

import { $config, handle, json } from "@notation/aws/lambda.runtime";
import { $dynamoClient } from "@notation/aws/dynamodb.runtime";
import { userTable } from "infra/api";

const userTableClient = $dynamoClient({
  table: userTable,
  allowedMethods: ["get", "batchGet"],
});

export const getTodos = handle.apiRequest(async () => {
  console.log(userTable); // ??
  const users = await userTableClient.get();
  return json(users);
});

export default $config({
  timeout: 5,
  memory: 64,
});

runtime output

import { handle, json } from "@notation/aws/lambda.runtime";
import { client as $dynamoClient } from "actual-client";

const userTableClient = $dynamoClient({ tableName: "actual-user-table-name" });

export const getTodos = handle.apiRequest(async () => {
  console.log(userTable); // ??
  const users = await userTableClient.get();
  return json(users);
});

infra output

import * as lambda from "@notation/aws/lambda";
import { userTable } from "infra/api";

const config = { 
  timeout: 5,
  memory: 64,
  policies: {
    userTablePolicy: userTable.executionPolicies(["get"])
  }
};

export const getTodos = lambda({
  fileName: "dist/runtime/todo/todos.fn/index.mjs", 
  handler: "getTodos", 
  ...config
});

// equivalient of:

// getTodos.addResource(
// 	new aws.lambda.LambdaFunction({ 
// 	  fileName: "dist/runtime/todo/todos.fn/index.mjs", 
// 	  handler: "getTodos", 
// 	  timeout: 5,
// 	  memory: 64,
// 	})
// );

// const lambdaRole = getTodos.addResource(
// 	new aws.lambda.LambdaExecutionRole({
// 		function: getTodos,
// 	});
// );

// getTodos.addResource(
// 	new aws.dynamodb.DynamoDbPolicyAttachment({
// 		role: lambdaRole,
// 		allowedMethods: ["get", "batchGet"],
// 	});
// );
  

macro runtime

// @notation/aws/dynamodb.runtime/dynamodb.ts
import { client } from "actual-client";

export function $dynamoClient (opts) {
  // remove access to unallowed methods etc.
  return client({});
}

macro plugin

// @notation/aws/dynamodb.runtime/dynamodb.macro.ts
export const $dynamoClient = {
  infra: $dynamoClientInfra, 
  runtime: $dynamoClientRuntime
};

// psuedo code
const $dynamoClientInfra = {
  updateConfig(args, argsNode, runtimeConfig) {
    const tableName = args.table.output.name;
    runtimeConfig.policies = runtimeConfig.policies ?? {};	
    runtimeConfig.policies[`${tableName}Policy`] = `${argsNode[0].identifier}.executionPolicies(${args.allowedMethods})`;
  }
}

const $dynamoClientRuntime = {
  imports: [`import { client as $dynamoClient } from "actual-client";`],
  buildArgs(args) {
    return [{ tableName: args.table.name }];
  },
};

[Happy0]

Your proposal looks good to me in general :).

Like you've identified in your 'questions' section, I'm also concerned about whether it should be possible to reference 'infrastructure resource' outputs in runtime code. There is a draw to this - for example, being able to reference a dynamically generated API gateway URL or passing data between the boundary of a build system via command line parameter to notation to modify the lambda's runtime behaviour without changing the code (such as passing the 'deployment environment name' letting the lambda know it's part of a test environment.)

Like you said though, this sort of thing requires a lot of care security wise (particularly around secrets like API keys, etc.) Where runtime code is concerned, if we're exposing resource outputs at runtime then these secrets would inevitably be 'static' data in the code itself (injected at notation compile time) which would make it visible to an adversary in the AWS lambda console when viewing or downloading code. Such data should be encrypted at rest and only accessible to people and lambda executions with sufficient AWS permissions.

It's a bit safer to pass such data at 'infrastructure compile time' via lambda environment variables if they're configured to be encrypted (and safer still to use Secrets Manager or KMS like you mentioned.)

I think I'm therefore comfortable with 2 options:

'Environment' configuration param in $config macro

Expose a 'Environment' param in the $config macro which is capable of evaluating resource outputs at infrastructure deploy time so that potentially sensitive data can't be passed to the runtime code directly. Under the hood in the iac layer this would populate the Environment param of the UpdateFunctionCodeCommand AWS API call (https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/lambda/command/UpdateFunctionCodeCommand/.)

I haven't finished writing my 'resource outputs' RFC yet, so I'm just sketching for now but you can imagine that an imported notation infrastructure resource might have a compile time type like Resource<T extends OutputSchema>:

export type OutputSchemaItem<T> = {
  valueType: z.ZodType<T>;
  presence: "required" | "optional";
}

export type OutputSchema = Record<string, OutputSchemaItem<any>>;

`type Resource<T extends OutputSchema> = ...`

So the runtime code wouldn't be at risk of accidentally making visible the secret parameters as it only sees a 'description' to be evaluated by the orchestration and deployment layers when imported (not the actual values.)

The 'environment' config could therefore accept either primitive values (static data) or OutputSchemaItem<any> which is dynamically evaluated to its real value at $config macro execution time, if that makes sense?

Example (sketch)

import { $config, handle, json } from "@notation/aws/lambda.runtime";
import { $dynamoClient } from "@notation/aws/dynamodb.runtime";
import { userTable, api } from "infra/api";

const userTableClient = $dynamoClient({
  table: userTable,
  allowedMethods: ["get", "batchGet"],
});

export const getTodos = handle.apiRequest(async () => {
  console.log(userTable); // ??
  const users = await userTableClient.get();
  return json(users);
});

export default $config({
  timeout: 5,
  memory: 64,
  environment: {
    apiUrl: api.outputs.url
  }
});

Runtime code has to use SSM / KMS / similar to access dynamic runtime config

This is certainly safer, but it probably doesn't fit with the expectations of users who are familiar with other IAC frameworks and it could be a bad user experience as requires much more setup so I'm not sure if it's acceptable.

[djgrant]

In agreement with everything here. A few more thoughts as this idea evolves.

I'm thinking along the same lines as you with the config/env example, but with a subtle difference.

Avoiding sensitive data being leaked

Infra resources should not have inputs or outputs parameters at build time i.e. the object we handle in our IDE doesn't autocomplete with an output param tempting you to inappropriately use it. The object is entirely innocuous. There is no way to leak any value from the infra resource at this stage.

import { userTable } from "infra/db";

export const getUsers = handle.apiGatewayRequest(() => {
  console.log(userTable); // { id: 'userTable-123456', type: 'dyanamodb-table' }
});

Although the above code sample is safe, it still isn't ideal because that infra import is going to add a lot of dead weight to the bundle.

The aim in this proposal (removing inputs/outputs params from resource objects) is less about avoiding leaking sensitive data and more about pushing developers into the pit of success. If we make it unobvious to use infra modules directly in runtime code, and obvious to use them within macros, the user should naturally find themselves doing the right thing.

In that case, after the runtime module is compiled, all infra module imports will be unused and removed by esbuild's dead code elimination. This creates an opportunity. If there are infra imports in the resulting bundle (which can be efficiently checked in the metafile that esbuild produces), we can infer that the user referenced an infra module in the runtime code, and an appropriate error message can be surfaced, prompting them to use a macro instead.

Accessing sensitive data

The lambda module compilation process would be something like this:

• compile lambda runtime modules
• compile infrastructure
• compile lambda infrastructure module, giving macros access to infra outputs

So to get the output URL, we'll have to extract it in the infra compile stage i.e. by retrieving the value from within the context of a macro.

As outputs does not exist on the resource object, we can't access it by property access. A macro can help:

import { userTable } from "infra/db";
import { $inlineOutput } from "@notation/aws/lambda.runtime"

// get an output from a resource (2nd param is typed with the union of non-sensitive property names)
const userTableName = $inlineOutput(userTable, 'name');

export const getUsers = handle.apiGatewayRequest(() => {
  console.log(userTableName); // 'userTable-123456'
});

I chose $inlineOutput to make clear that this macro produces a literal value in the code.

For sensitive values, this would not be allowed, and the user would be expected to use a secrets service, although we'll really help them with this:

import { userTable } from "infra/db";
import { $kmsClient } from "@notation/aws/lambda.runtime";

// haven't thought about this api at all, just a rough sketch
const secrets = $kmsClient.fromResources({ userTable });

const userFact = getUserFact(secrets.userTable.userKey);

export const getUsers = handle.apiGatewayRequest(() => {
  console.log(userFact);
});

[Happy0]

Sounds good to me :).

It has the advantage over what I suggested of expanding the macro system that's already being introduced with the $dynamoClient imports and makes that the consistent way to bridge infrastructure and runtime concerns (rather than introducing another way.)

I also like taking advantage of the secret type fields that are already in the resource 'schema' types.