RFC: Allow resources to return outputs and use other resource outputs for configuration

[Happy0]

Definitions

Resources

An infrastructure entity which is managed (created, configured, updated and deleted) by Notation. In this RFD, I am referring to the high level ‘resource’ exposed to the user from the top level API package (‘aws’ in the case of notation AWS resources) via a function based interface. Examples:

• An AWS API gateway resource to create a HTTP API.
• An AWS event bridge resource allows a user to schedule a function to be executed periodically.

Resource Groups

A resource is often a composition of other infrastructure resources managed by notation internally into ‘resource groups.’ For example, an API gateway route handler internally constructs an AWS lambda service ‘lambda’ function resource and sets up the required permissions to allow the API gateway resource to execute the function as dependencies. See #5 for more details on how Resource Groups work conceptually.

‘ResourceGroup’ is the current return type of Notation resources at the API layer, encapsulating all the ‘inner’ resources constructed to build the notation resource ( https://github.com/notationhq/notation/blob/main/packages/core/src/orchestrator/resource-group.ts#L9 ). A ResourceGroup is sometimes passed to other resources to wire together resources that are dependent on each other - for example an API gateway group passed to a ‘API route’ resource.

Outputs

Meaningful data returned by a resource on successful deployment. For example, an API gateway resource may return the dynamically generated Base URL of the newly deployed HTTP API. An SQS queue resource might return the queue URL (which can be used to enqueue new entries.)

Orchestration Time - The stage where notation builds a representation of all the resources to deployed, making it possible to determine which resources need to be deployed as a dependency of others.

Deploy Time - When notation deploys all the resources defined by the notation app.

API Layer - The ‘top level’ at which the user interacts with notation to use predefined resources (e.g. construct an API gateway, etc.)

IAC Layer - The low level resource definitions that are capable of orchestrating the underlying provider (e.g. AWS.)

Aims

• To allow resources to return outputs when deployed.
• To allow resources to receive inputs (configuration) from the output values of other resources at deployment time.
• To allow runtime code to securely access infrastructure output values.
• Modularity: Resources can receive these inputs even if importing the resource from another file, folder, package or external package.
• Type safety: the outputs for a resource should be known at compile time via types, resulting in a compilation error if the user tries to access an output not returned from a resource in code.

Example Use Cases

• Storing application configuration in Parameter Store / SSM / similar. For example, storing dynamically generated API gateway resource base URL, SQS queue URLs, as the ‘value’ input for configuration storage resources.
• Printing global ‘notation stack’ output at deployment time in the terminal (such as API routes, URLs, etc.)
• Libraries / modules / packages: possible to create notation modules / packages which opaquely create some resource and return some values relevant to the resource. For example, a ‘docker image’ resource that returns the generated image’s ID.

Proposed Approach

The proposed approach requires two API level considerations:

• How we return outputs from notation resources
• How we ‘wire’ outputs from resources to inputs of notation resources

Outputs

Treating ResourceGroup as the entry point to a resource that the user will interact with, I propose we add a new ‘outputs’ field that the user can access.

New ResourceGroup Type

export type ResourceGroupOutput<T> = {
  parentId: number,
  valueType: z.ZodType<T>;
  presence: "required" | "optional";
  sensitive?: true;
  hidden?: true;
  ignore?: true;
}

export abstract class ResourceGroup {
  type: string;
  id: number;
  dependencies: Record<string, number>;
  outputs: Record<string, ResourceGroupOutput<any>>;
  config: Record<string, any>;
  resources: BaseResource[];

Note: parentId allows the orchestrator to work out what ResourceGroup an output came from when it is used as an input. It does not necessarily need to be passed in at construction as it could be calculated in ResourceGroup’s constructor.

Inputs

Until now, a configuration input to a ResourceGroup is passed in as a static value (or computed expression) by the user (e.g. the ‘name’ config field here is a string value):

const todoApi = api({ name: "todo-api" });

We need to be able to represent in the type system that input can dynamically come from another resource’s output at runtime.

I propose we introduce a new Notation type that is capable of representing either a value or a value which will be wired in from another resource group’s output during deployment for each config field.

Consider the following type

export type NotationInput<T> = T | ResourceGroupOutput<T>

This isn’t quite sufficient because it is incapable of representing whether the ‘presence’ (required or optional) between the target output and input are compatible. We can redefine the NotationInput type to impose compatibility by defining NotationInput in the following way:

export type RequiredNotationInput<T> = T | ResourceGroupOutput<T> & { presence: "required" }
export type OptionalNotationInput<T> = T | ResourceGroupOutput<T> & { presence: "optional" } | undefined | null
export type NotationInput<T, X extends "required" | "optional"> = X extends "required" ? RequiredNotationInput<T> : OptionalNotationInput<T>

Transformations

I propose the runtime resolved value of a ‘ResourceGroupOutput’ should be transformable at runtime (akin to how Promises are ‘thenable’ to transform to new promises) as there will be cases where the user doesn’t want to use the exact output value of the resource. Example:

export const transform: <T, X extends "required" | "optional", R>
  (input: NotationInput<T, X>, transform: ((resolved: T) => R)) => NotationInput<R, X> = ...

We could consider allowing async operations by allowing the ‘transform’ parameter to return a Promise<R> type.

Formalising Notation Resource Interfaces

At present, ResourceGroup has the following fields:

  type: string;
  id: number;
  dependencies: Record<string, number>;
  config: Record<string, any>;
  resources: BaseResource[];

The group’s internal resources can be accessed via the following function:

findResource<T extends new (...opts: any[]) => BaseResource>

This is sometimes used to find a resource in the group required to create another low level resource such as the API gateway required to bind a api gateway route resource to:

export const route = (
  apiGroup: ReturnType<typeof api>,
  method: string, // todo: http methods only
  path: `/${string}`,
  handler: ApiGatewayHandler,
) => {
  const apiResource = apiGroup.findResource(aws.apiGateway.Api)!;

This resource is then passed to another IAC level resource which accesses its ‘output’ fields at provision time to fetch the necessary data to continue with deployment (via a 'dependencies' field.)

I propose that the details of the internal resources a notation resource (or resource group) is made up of should be opaque (private fields) and only the explicit ‘outputs’ should be visible to be referenced either as ‘implicit’ dependencies to other resources as above or as user defined ‘input values’ to another resource.

To this end, perhaps the ‘ResourceGroup’ type should be renamed to ‘NotationResource’ or similar.

Diagram

Before

flowchart LR
  subgraph API
   	 API_Stage --> API_Gateway  
  end

    subgraph API_Route_1
	API_Gateway_Route_1 --> API_Gateway
  end

    subgraph API_Route_2
	API_Gateway_Route_2 --> API_Gateway   	 
    end
    
    subgraph Function
   	 Integration_Lambda_A --> API_Gateway
   	 Permission_Lambda_A --> API_Gateway
	Integration_Lambda_A --> Lambda_A
	Permission_Lambda_A --> Lambda_A
	Role_Attachment --> Role
	Lambda_A --> Role_Attachment
  end

    API_Gateway_Route_2 --> Integration_Lambda_A
    API_Gateway_Route_1 --> Integration_Lambda_A

Loading

After

flowchart LR


  subgraph API
        API_Stage --> API_Gateway


        subgraph API_OUTPUTS
            API_GATEWAY_BASE_URI
            API_GATEWAY_ID


        end
  end


    subgraph API_Route_1
    API_Gateway_Route_1 --> API_GATEWAY_ID
  end


    subgraph API_Route_2
    API_Gateway_Route_2 --> API_GATEWAY_ID      
    end
   
    subgraph Function
        subgraph FUNCTION_OUTPUTS
            FUNCTION_ARN


        end


    Role_Attachment --> Role
    Lambda_A --> Role_Attachment
  end


  subgraph API_GATEWAY_INTEGRATION
        Integration_Lambda_A --> API_GATEWAY_ID
        Permission_Lambda_A --> API_GATEWAY_ID


        Integration_Lambda_A --> FUNCTION_ARN
        Permission_Lambda_A --> FUNCTION_ARN


        subgraph INTEGRATION_OUTPUTS
            INTEGRATION_ID
        end
  end


    API_Gateway_Route_2 --> INTEGRATION_ID
    API_Gateway_Route_1 --> INTEGRATION_ID


  subgraph SSM_PARAM
    API_URL --> API_GATEWAY_BASE_URI
  end

Loading

Advantages

• Better modularity: Notation Resources have an explicit interface, allowing internal details to be changed without risking breaking clients who are accessing ‘inner resources’ and their fields, etc.
• We may be able to get rid of ‘requireDependencies’ as our output and input types are tagged with their parent resource ID. It could be the job of the orchestrator to calculate the dependency graph based on these.

Disadvantages

• May have to consider whether we represent the outputs in the visualisation CLI tool output as above as it may obscure the reality of the deployed system due to indirection. Perhaps we could try and hide that in the visualisation output.

UX Considerations

Where appropriate, I think we should continue to expect entire ‘NotationResource’ (or ‘ResourceGroups’) as parameters where there are implicit dependencies (such an API route expecting an API resource param) between resources at the API layer.

This is more type safe than expecting the user to pass the ‘raw’ output values required as inputs and acts as a level of documentation.

I think we should expect notation resources to return types that extend the raw ‘NotationResource’ type (e.g. API gateway would return a type named APIGateway even if it was just a type alias for NotationResource.)

However, I think the ‘iac layer’ should accept the ‘low level’ fields it requires rather than introspecting IAC level resources. This allows us to keep the API layer opaque since a NotationResource does not need to expose its internal IaC resources. Additionally, the IaC layer does not need knowledge of the API layer.

Implementation Challenges

Orchestration

Low level ‘resources’ must be capable of accepting the new ‘NotationInput’ types as values where they match a schema field’s field name and presence values. They must be able to represent in the orchestration graph which resource owns these output values (by resource ID.)

This may be challenging as the type level code which transforms a resource schema to the concrete expected types is already quite complex.

We must also be able to detect infinite dependency loops between chains of inputs and outputs and raise an error.

Deployment

We must be able to create and follow a deployment order strategy which deploys resources in the right order to allow outputs to be wired to the inputs of other resources.

We may need to change our deployment strategy as currently resources accept other resources as dependencies rather than their outputs.

[djgrant]

Really appreciate the detailed exploration on this @Happy0. Still wrapping my head around the broader implications so a few points and clarifying questions to start.

To allow resources to receive inputs (configuration) from the output values of other resources at deployment time.

We do this in the IaC layer. It would be good to understand what kind of use cases need this in the API layer.

I propose that the details of the internal resources a notation resource (or resource group) is made up of should be opaque (private fields) and only the explicit ‘outputs’ should be visible to be referenced either as ‘implicit’ dependencies to other resources as above or as user defined ‘input values’ to another resource.

Agree.

To this end, perhaps the ‘ResourceGroup’ type should be renamed to ‘NotationResource’ or similar.

Good suggestion

I propose we introduce a new Notation type that is capable of representing either a value or a value which will be wired in from another resource group’s output during deployment for each config field.

I've so far resisted this because to avoid more complex code when unpacking types e.g. https://www.pulumi.com/docs/concepts/inputs-outputs/. This kind of code feels like it very much belongs in the IaC layer. But there are inevitably use cases in the API layer. Is it possible to support them without losing the current feel of the framework.

Diagram

I like this structure. One point of clarification though: the NotationResource, in my mind at least, represents the resource declared in the API layer. API integration, for example, makes sense as a logical grouping, but is not a resource declared in the API layer. Perhaps this could be called an InferredNotationResource.

We may be able to get rid of ‘requireDependencies’ as our output and input types are tagged with their parent resource ID

The key advantage of this is less for the orchestrator and more for contributors (or developers who use the IaC layer directly). Having the dependencies defined statically, constrains them to only connecting infra resources that are compatible.

May have to consider whether we represent the outputs in the visualisation CLI tool output as above as it may obscure the reality of the deployed system due to indirection. Perhaps we could try and hide that in the visualisation output.

For visualisation, I prefer your grouping. I also like seeing which output values are passed around but it's probably a bit much to take in in on the first glance. Useful when drilling down into the detail. Not a big problem transforming the graph for visualisation though.

What do you mean by indirection?

Where appropriate, I think we should continue to expect entire ‘NotationResource’ (or ‘ResourceGroups’) as parameters where there are implicit dependencies (such an API route expecting an API resource param) between resources at the API layer.

I agree. This replaces uncertainty (which param am I passing in here?) with a guarantee (just give us the whole resource and we'll figure it out).

However, I think the ‘iac layer’ should accept the ‘low level’ fields it requires rather than introspecting IAC level resources.

Do you mean not having to use findResource?

We must be able to create and follow a deployment order strategy which deploys resources in the right order to allow outputs to be wired to the inputs of other resources.

Also need to explore if a new approach to outputs opens us up to not being able to generate the orchestration graph at orchestration time.

[Happy0]

We do this in the IaC layer. It would be good to understand what kind of use cases need this in the API layer.

I think it's probably worth holding off on implementing this until we feel its absence... It could be that the 'dependencies' approach to wire resources together that Notation already has is good enough for most cases.

Certainly often when I've used 'outputs' in other IAC frameworks in the past it tends to be to deal with things that should probably be an abstraction (such as setting up a dynamo DB stream lambda handler and having to get the "Stream ARN" output attribute of the dynamo stream resource to configure the lambda handler resource with the 'input event source' set to the raw ARN.

I've also used it for passing output params (such as an SQS queue URL that may be dynamically named based on staging environment) to a lambda environment variable. However, we already have that use case covered a bit in #17 .

Perhaps this RFC is trying to address a problem that doesn't / needn't exist and is too hasty.

But there are inevitably use cases in the API layer. Is it possible to support them without losing the current feel of the framework.

I know what you mean. I don't particularly like the 'feel' of polluting the API layer input types with this complexity either.

I can see two high level alternatives that might be worth exploring:

Imperative execution model

The infrastructure code is the provisioning code - there is no indirection between the provisioning and deployment phases. The code you write for your infrastructure is the same code that executes the deployment.

const todoApi = await api({ name: "todo-api" });
const todoRouter = router(todoApi);

const apiUrl: string = todoApi.outputs.baseUrl

Here each resource returns a Promise<NotationResource> type that completes when the resource has been deployed / updated / read. The use of Promise.then defines the dependencies between resources.

I'm sure whether this is at all viable or not will be more apparent to yourself since you've been knee deep in this side of things for much longer!

My hunch is this is probably a bad idea but I can't really justify why. We'd certainly lose the ability to visualise the resource graph, I know that much.

Higher Level 'wiring' API

NotationGroup types still return the ResourceGroupOutput types as above, but we don't pollute our resource configs to expect a NotationInput type directly. The configs would still accept the primitive values (string, number, object, etc) but for situations where we want these fields to be the output of another resource output there is a higher level function which wires things together for us. Example:

import { api, router } from "@notation/aws/api-gateway";
import { getTodos, getTodoCount } from "runtime/todo/todos.fn";
import { ssmParam } from "@notation/aws/ssm-param"

const todoApi = api({ name: "todo-api" });
const todoRouter = router(todoApi);

todoRouter.get("/todos", getTodos);
todoRouter.get("/todos/count", getTodoCount);

withOutputs(todoApi, (apiGatewayOutputs: any) => (
    ssmParam({
    "type": "String",
    "Name": "APIGateWayURL",
    "Value": gatewayOutputs.apiUrl
 }
)))

The key advantage of this is less for the orchestrator and more for contributors (or developers who use the IaC layer directly). Having the dependencies defined statically, constrains them to only connecting infra resources that are compatible.

Yeah... That makes sense to me at the IaC layer. At the API level I don't think having to use findResource to work with the opaque 'ResourceGroup' is easy to work with at the moment - it's kind of a 'hidden' interface. The suggestion in this RFC was an effort to make the interface for resources more 'discoverable' / obvious - but I do agree this would be a bit of a loss at the IaC level.

Perhaps something in the middle would be that our NotationResource types extend ResourceGroup and are expected to call ResourceGroup::add internally for each 'raw resource' but also expect these resources in the constructor and set them as class fields that are accessible.

What do you mean by indirection?

Oh, just that there isn't a line indication which underlying resource the output comes from directly... We just know from the diagram that it comes from one of the sources - which at present we know exactly where it comes from.

Do you mean not having to use findResource?

I think so yeah... I discussed this a bit above, I suppose :)

[djgrant]

Picking up on:

Modularity: Resources can receive these inputs even if importing the resource from another file, folder, package or external package.

A big use case for this would be a stack deployed by Notation (e.g. an API) that is consumed by another app deployed elsewhere. That app might be in the same monorepo. Ideally I'd like type interop between the backend API and the front-end app, even if though they are deployed elsewhere.

Putting to one side versions and coordinating deploys, let's ask: how do I get the outputs from my API in my front-end app?

I'd like to put forward a proposal to use NPM packages as a way to expose an interface from a Notation stack to another app (or another Notation stack).

NPM packages have a few advantages:

• They're native to the ecosystem. We don't need to convince anyone to adopt a new system.
• They're versioned (although backwards compat is a whole other RFC)
• They can be easily deployed to public and private registries
• They can be shared within a monorepo without publishing
• They have an entry point (package.json main field)

How might this work?

1. Build your stack inside an npm package
2. Expose outputs via an entry point e.g. index.ts
3. Notation will compile your entry point to a module containing just the exposed outputs
4. If publishing, you set the files field in package.json to only include the compiled outputs file
5. Import the package in other projects

That provides a way to consume stacks in any other JavaScript projects. To provide type interop Notation could provide decorators. For example:

// packages/stacks/user-api/index.ts
import { apiClient } from "@notation/aws/clients";
import { userApi } from "./api";

export const userApiClient = apiClient({ api: userApi });

And then in your app:

// packages/apps/dashboard
import { userApiClient } from "@stacks/user-api";

// this would be typed based on the API routes
const users = await userApiClient.get("/users");