RFC: Validation

[misterjoshua]

Ref #445 for the structure of this RFC

Description of the proposal

There should be a consistent way to validate the structures of inputs and outputs for AWS lambdas.

Name of the core utility (and consequently the folders that contain the libraries):

• packages/validation

Justification

The "Garbage in, garbage out" principle suggests that we should validate our inputs to prevent "garbage in," and for Lambda, those inputs include events passed to the lambda handler. A short, informal survey of GitHub projects using the CDK's NodejsFunction found that 90% of the studied projects hand-code their validation, indicating little consensus on any singular tool to validate inputs.

Additionally, the Java and Python toolkits saw fit to add validation of not only lambda input but lambda responses.

Validation support should be added to work toward feature parity between the AWS Lambda Powertools libraries and to offer developers a good alternative to hand-coding their validation.

Goals

• Validate incoming events and responses
• Unwrap events before validation applies
• Built-in envelopes to unwrap popular event source payloads

Proposed API

Installation

Yarn

yarn add @aws-lambda-powertools/validation

Npm

npm install @aws-lambda-powertools/validation

Usage

Lambda Handler

import { Validation, Envelope } from "@aws-lambda-powertools/validation";

interface GreetingRequest {
  readonly name: string;
}

interface GreetingResponse {
  readonly message: string;
}

async function handlerBase(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
  return {
    message: `Hello ${request.name}`,
  };
}

export const handler = Validation.handler(handlerBase, {
  envelope: Envelope.jmesPath("detail"),
  // Schemas are JSON Schema
  inboundSchema: {
    type: "object",
    required: ["name"],
    properties: {
      name: { type: "string" },
    },
  },
  outboundSchema: {
    type: "object",
    required: ["message"],
    properties: {
      message: { type: "string" },
    },
    additionalProperties: false,
  },
});

Ad-hoc validation

import { Logger } from "@aws-lambda-powertools/logger";
import { Validator } from "@aws-lambda-powertools/validation";

const logger = new Logger();

interface Person {
  readonly name: string;
}

const validator = new Validator<Person>({
  schema: {
    type: "object",
    required: ["name"],
    properties: {
      name: { type: "string" },
    },
  },
});

try {
  const person: Person = validator.map(JSON.parse(someInput));
  logger.info(`Person's name is ${person.name}`);
} catch (e) {
  logger.error(`Failure: ${e}`);
}

Survey Results

Obtained by searching GitHub code for NodejsFunction in TypeScript projects. The repositories are from the first twenty results, sorted by "best match." These cases are good enough evidence for me, but perhaps the community can suggest a better way to get this information.

Validation	URL
JOI	https://github.com/Mrdev1ce/rs-shop-be
Hand-coded	https://github.com/jontiefer/space-finder-backend
@softchef/lambda-events	https://github.com/taylorr-liu/EX7
Hand-coded	https://github.com/ryands17/passwordless-auth
Hand-coded	https://github.com/josh-hill-gene/cdk-serverless-react
Hand-coded	https://github.com/zakiafada32/swj
Hand-coded	https://github.com/martzcodes/blog-cdk-openapi
Hand-coded	https://github.com/Shridharbhandar/CDK-Samples
Hand-coded	https://github.com/roman-boiko/apigw-workshop
Hand-coded	https://github.com/aws-samples/amazon-qldb-product-management
Hand-coded	https://github.com/cjbatin/GetStatDashboard
Hand-coded	https://github.com/jforge/iac-samples
Hand-coded	https://github.com/okaharuna/e-payment
Hand-coded	https://github.com/balancer-labs/pools-api
Hand-coded	https://github.com/BertoDBQ/space-finder-backend
Hand-coded	https://github.com/XiaozhouCui/space-finder-backend
Hand-coded	https://github.com/akira393/2021-typescript-ddd
Hand-coded	https://github.com/JonathanTurnock/aws-cabiinet
Hand-coded	https://github.com/aws-samples/az-fail-away
Hand-coded	https://github.com/enricoschaaf/analytics

[willfarrell]

Engines:

• https://ajv.js.org/
• https://github.com/ebdrup/json-schema-benchmark (other json schema projects)
• https://joi.dev/

Existing lambda middleware:

• https://github.com/middyjs/middy/tree/release/3.x/packages/validator [ajv]
• https://github.com/willfarrell/middy-ajv [ajv]
• https://www.npmjs.com/package/middy-sparks-joi [joi]

[flochaz]

Thanks a lot for those inputs ! We will look into it

[ijemmy]

@misterjoshua Thanks a lot. This is very nice RFC and rich in details.

We plan to catch up with Python feature so this is in our roadmap. The team is focusing in fixing all outstanding issues on 3 core utilities to have Production Ready version. So it may take a while to implement this. However, I would like to use this chance to discuss the feature API so we have a clear spec, ready for implementation.

This feature is comparable to the Validation utility in Python version. The main difference is that Python has top-level function decorator, but TypeScript doesn't have that. This is the same challenge we had in core utility design(tracer, logger, metrics).

Let's explore the alternatives so we can weight pros & cons:

Validation class

@misterjoshua What do you think of this alternative?

Usage 1: Middy middleware

import { Validation, Envelope, validateSchema } from "@aws-lambda-powertools/validation";


const validation = new Validation({
  envelope: Envelope.jmesPath("detail"), // Optional: only if we want to validate a part of it
  inboundSchema: {...},
  outboundSchema: {...}, //optional
});

// Copied from proposal
//....
async function handlerBase(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
  return {
    message: `Hello ${request.name}`,
  };
}
//...

export const handler = middy(handlerBase)
  .use(validateSchema()
  .use(..); // can chain with Logger's injectContext() or any other utilities

The advantage is that it's easier to chain multiple utility features. In your proposal, if we also want to use injectLambdaContext(logger), we will need to wrap Validation object with another function (or class) like this:

export const handler = Tracer.captureLambdaHandler(Logger.injectContext(
  Validation.handler(handlerBase, {...});
)));

Usage 2: Class decorator

We can follow the same pattern of core utilities.

import { Validation } from `....`;
 
const validation = new Validation({
  envelop: ...;
  inboundSchema: ...;
  outboundSchema: ...;
});

class Lambda implements LambdaInterface {
    // Decorate your handler class method
    @validation.validateHandler()
    public async handler(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
        /* ... */
    }
}

Usage 3: manual

This is where I see no clear winner. Python version have only a validate() function.

The Validator class like in your ad-hoc example looks good to me. Another alternative is function based like in Python

const validationOption = {
  envelop: ...;
  inboundSchema: ...;
  outboundSchema: ...;
};

try {
  const person: Person = validate<Person>(JSON.parse(someInput), validationOption);
  logger.info(`Person's name is ${person.name}`);
} catch (e) {
  logger.error(`Failure: ${e}`);
}

That being said using Validator class also makes a lot of sense to me too. I'm wondering if I miss any pros & cons here.

Other points to consider:

1. Should we make outboundSchema required parameter?
2. If inbound validation fails, it fails fast by throwing an Error describing that there is an incorrect format.
   • Should we have an option to control the level of info we disclose here?
3. If outbound validation fails, returns a 500 error.
   • How do we log the error in this case? (we can use Logger but that means another dependency) Should we provide an optional log function with console.error() as a default?
   • Or we let users handle this themselves with manual option
4. middy has already had a few Validator middlewares . The only difference I see is JMES support. If we provide an extractor for JMESPath (like in Python) We could reuse existing ones. This makes me concerned if we are reinventing a wheel here. (But the same could be said for all other utilities)

[misterjoshua]

Usage 1: Middy middleware

@ijemmy I like the middleware idea - it's familiar to many node devs. I have a few thoughts on the example above:

• I presume that validateSchema would have a parameter that accepts your Validation instance?
• Also, handlerBase wants GreetingRequest - would the validateSchema middleware strip the envelope?
• I wonder if there's an opportunity here to validate that the envelope is right. e.g., to check that a lambda meant to be an Event Bridge Rule Target received Event Bridge events and not SQS, SNS, or API Gateway events.
• And I wonder if it's possible to provide an Envelope type that allows users to create a multi-purpose lambda handler - one handler that works for any AWS event type that includes payload data. (SNS, SQS, Event Bridge, or Kinesis Streams for example.) Users could then create lambdas that don't need to pay much attention to identifying and unpacking the structure of the envelopes for each payload.
• Finally, I wonder whether Middy middleware can flat map records from batch event types. If it can, this plays well with the story of creating a multi-purpose lambda function.

What about something like this? (EDIT: I have added some discussion near the bottom of this comment about separating Validation from Envelope as another decorator and middleware to be applied before validation.)

import { Validation, Envelope, validateSchema } from "@aws-lambda-powertools/validation";

// Composable, multi-purpose envelope-stripping example.
const envelope = Envelope.multiPurpose({
  envelopes: [
    // Accepts the given detail type and unwraps `details`
    Envelope.awsEventBridgeEvent({ detailType: 'Detail Type Name' }),
    // Flat maps SQS event `Records[*].body` to parsed json and feeds each
    // through `handlerBase` when the entire event is valid.
    Envelope.awsSqsEvent({ jsonPayload: true, flatMapRecords: true }),
    // Same idea as SQS, but for SNS and `Records[*].Sns.Message`
    Envelope.awsSnsEvent({ jsonPayload: true, flatMapRecords: true }),
  ],
});

const validation = new Validation({
  envelope: envelope, // Replaced multi-purpose envelope example
  inboundSchema: {...},
  // Commented out for the sake of this example:
  // outboundSchema: {...},
});

// Multi-purpose handler - it doesn't need to know how it's getting GreetingRequest, whether it's from
// SNS, SQS, or Event Bridge.
async function handlerBase(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
  return {
    message: `Hello ${request.name}`,
  };
}

export const handler = middy(handlerBase)
  .use(validateSchema(validation)) // This example assumes Middy can flat map SQS/SNS batch records.
  .use(...);

Usage 2: Class decorator

This looks good. I like the decorator syntax. Here's an example of the same multi-purpose handler as above:

import { Validation, Envelope } from "@aws-lambda-powertools/validation";

// Multi-purpose envelope unwrapping example (same as previous example)
const envelope = Envelope.multiPurpose({
  envelopes: [
    Envelope.awsEventBridgeEvent({ detailType: 'Detail Type Name' }),
    Envelope.awsSqsEvent({ jsonPayload: true, flatMapRecords: true }),
    Envelope.awsSnsEvent({ jsonPayload: true, flatMapRecords: true }),
  ],
});

const validation = new Validation({
  envelope: envelope, // Replaced multi-purpose envelope example
  inboundSchema: {...},
  // Commented out for the sake of this example:
  // outboundSchema: {...},
});

class Lambda implements LambdaInterface {
    // Decorate your handler class method
    @validation.validateHandler()
    public async handler(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
        /* ... */
    }
}

Usage 3: manual

This is where I see no clear winner. Python version have only a validate() function.

I agree. Validator.map departs from the "validate function" approach in both the Python and Java libs. To maintain consistency, your syntax looks better. I especially like the idea that the validate function returns typed data from the JSON.parse(). We encounter a need for this type of function when working with data returned from the AWS SDK. (SecretsManager secrets in JSON format and DynamoDB Document Client items come to mind.)

Other points to consider:

1. Should we make `outboundSchema` required parameter?

I think not. For backend lambdas, the output isn't always important. (i.e., Lambdas handling SQS, SNS, or Event Bridge Rule Targets.)

2. If inbound validation fails, it fails fast by throwing an `Error` describing that there is an incorrect format.
   * Should we have an option to control the level of info we disclose here?

Yes, I think so. For API Gateway, I can envision a scenario where I want to create an entirely custom error response with a particular HTTP status code. Perhaps Validator can accept a validationErrorHandler option to run when validation fails. Example:

async function validationErrorHandler(error) {
  return {
    statusCode: 418, // HTTP 418 I'm a teapot
    message: 'My custom message here',
  };
}

const validation = new Validation({
  inboundSchema: {...},
  outboundSchema: {...},
  validationErrorHandler: validationErrorHandler,
});

// Provide `validation` to middleware or use the decorator approach.

3. If outbound validation fails, returns a 500 error.
   
   * How do we log the error in this case? (we can use `Logger` but that means another dependency) Should we provide an optional log function with `console.error()` as a default?

Given that Tracer, Logger, and Metrics already depend on Commons, could we add something like ClassThatLogs) to Commons? If we do, we can optionally accept ClassThatLogs in Validator with a fallback to some default implementation.

const logger = new Logger(...);

const validation = new Validation({
  inboundSchema: {...},
  outboundSchema: {...},
  logger: logger,
});

   * Or we let users handle this themselves with manual option

If we put ClassThatLogs in Commons, the user can provide a custom implementation.

4. `middy` has already had a few Validator middlewares . The only difference I see is JMES support. If we provide an extractor for JMESPath ([like in Python](https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/jmespath_functions/)) We could reuse existing ones. This makes me concerned if we are reinventing a wheel here. (But the same could be said for all other utilities)

I see value in having the decorator version of the validation even if we don't add a middy middleware due to overlap in their ecosystem... But I don't see how adding a middleware for Middy would add much complexity if we're already adding a decorator version, since the majority of the logic would live in Validator, no?

EDIT: I was just thinking that at a certain point, I'm not sure if the idea of multi-purpose lambdas is just a validation story - it could be two stories: Validation and Extraction. I figure it should be possible to separate Validation and Extraction into two distinct features. Extraction could just as well be its own middleware & decorator, perhaps centred around the concept of Envelope. I imagine that could look like this:

// Composable, multi-purpose envelope-stripping example.
const envelope = Envelope.multiPurpose({
  envelopes: [
    // Accepts the given detail type and unwraps `details`
    Envelope.awsEventBridgeEvent(),
    // Flat maps SQS event `Records[*].body` to parsed json and feeds each
    // through `handlerBase` when the entire event is valid.
    Envelope.awsSqsEvent({ jsonPayload: true, flatMapRecords: true }),
    // Same idea as SQS, but for SNS and `Records[*].Message`
    Envelope.awsSnsEvent({ jsonPayload: true, flatMapRecords: true }),
  ],
});

const validation = new Validation({
  inboundSchema: {...},
  // Commented out for the sake of this example:
  // outboundSchema: {...},
});

// Multi-purpose handler - it doesn't need to know how it's getting GreetingRequest, whether it's from
// SNS, SQS, or Event Bridge.
async function handlerBase(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
  return {
    message: `Hello ${request.name}`,
  };
}

export const handler = middy(handlerBase)
  .use(extractFromEnvelope(envelope)) // This example assumes Middy can flat map SQS/SNS batch records.
  .use(validateSchema(validation))
  .use(...);

Here's use case 2:

// Composable, multi-purpose envelope-stripping example.
const envelope = Envelope.multiPurpose({
  envelopes: [
    // Unwraps `details`
    Envelope.awsEventBridgeEvent(),
    // Flat maps SQS event `Records[*].body` to parsed json and feeds each
    // through `handlerBase` when the entire event is valid.
    Envelope.awsSqsEvent({ jsonPayload: true, flatMapRecords: true }),
    // Same idea as SQS, but for SNS and `Records[*].Message`
    Envelope.awsSnsEvent({ jsonPayload: true, flatMapRecords: true }),
  ],
});

const validation = new Validation({
  inboundSchema: {...},
  // Commented out for the sake of this example:
  // outboundSchema: {...},
});

class Lambda implements LambdaInterface {
    // Decorate your handler class method
    @envelope.extractForHandler()
    @validation.validateHandler()
    public async handler(request: GreetingRequest, context: lambda.Context): Promise<GreetingResponse> {
        /* ... */
    }
}

I'm happy to open another RFC focused on Extraction, if this is something that could work.

[willfarrell]

Middy takes this approach of separating normalization (envelope/Extraction) and validation. See AWS Event support list, we're just missing API Gateway WebSocket events which will be its own middleware at some point. I haven't heard of anyone requesting the ability to flatMapRecords, but it could be added pretty easily if someone needed it.

If AWS provided the JSON schemas for each event in the documentation, that would be super helpful. I've thought about writing these myself, but i'd just be reverse engineering from the samples and may not be accurate.

[ijemmy]

@misterjoshua

I presume that validateSchema would have a parameter that accepts your Validation instance?

That's right.

Also, handlerBase wants GreetingRequest - would the validateSchema middleware strip the envelope?

I'm inclined to have two different middlewares for different tasks.

I wonder if there's an opportunity here to validate that the envelope is right. e.g., to check that a lambda meant to be an Event Bridge Rule Target received Event Bridge events and not SQS, SNS, or API Gateway events.

Is it something similar to this Built-in envelops in Python version?

If yes, this should be included in the spec.

And I wonder if it's possible to provide an Envelope type that allows users to create a multi-purpose lambda handler - one handler that works for any AWS event type that includes payload data. (SNS, SQS, Event Bridge, or Kinesis Streams for example.) Users could then create lambdas that don't need to pay much attention to identifying and unpacking the structure of the envelopes for each payload.

Can you tell me more about your business use cases?

I haven't seen many use cases of consuming the same data from different AWS source.

Finally, I wonder whether Middy middleware can flat map records from batch event types. If it can, this plays well with the story of creating a multi-purpose lambda function.

I think this is out of scope for Validation. I would go with an extra line of _.flatMap(). If you want it in middleware, you can create a custom middle ware with that.

Given that Tracer, Logger, and Metrics already depend on Commons, could we add something like ClassThatLogs) to Commons? If we do, we can optionally accept ClassThatLogs in Validator with a fallback to some default implementation.

That could be a good option. The price we pay is leaking abstraction between modules.

I see value in having the decorator version of the validation even if we don't add a middy middleware due to overlap in their ecosystem... But I don't see how adding a middleware for Middy would add much complexity if we're already adding a decorator version, since the majority of the logic would live in Validator, no?

No, it won't add much complexity.

[ijemmy]

Middy takes this approach of separating normalization (envelope/Extraction) and validation. See AWS Event support list, we're just missing API Gateway WebSocket events which will be its own middleware at some point. I haven't heard of anyone requesting the ability to flatMapRecords, but it could be added pretty easily if someone needed it.

If AWS provided the JSON schemas for each event in the documentation, that would be super helpful. I've thought about writing these myself, but i'd just be reverse engineering from the samples and may not be accurate.

Wow, this is pretty good. I'm going to use that in my next project :)

Regarding JSON schemas, let me pass this feedback to the Lambda team.

[bestickley]

I'd like to add another validation engine to be considered: zod. IMO it has the best TypeScript DX between ajv and joi.

EDIT: When I made the above comment, I was not aware of the parser utility tracked in #1334. Personally, I probably won't use the validation package as the parsing validation does validation and parsing and I'd rather have them all as one.