Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Lambda json parsing #328

Merged
merged 12 commits into from
Apr 23, 2024
Merged

Fix: Lambda json parsing #328

merged 12 commits into from
Apr 23, 2024

Conversation

ben-taussig-solo
Copy link
Contributor

@ben-taussig-solo ben-taussig-solo commented Apr 11, 2024

Description

Summary of bug

  • The bug occurs when unwrapAsApiGateway is enabled in the lambda filter
    • in this case, we attempt to convert a json payload returned by the upstream lambda into a response forwarded to the downstream client
  • in this case, we assumed that we'd receive a payload w/ the form:
{
  "multiValueHeaders": {
    "x-header-foo": [
      "bar1",
       "bar2"
    ]
  }
}
  • instead, we choked on a payload like this (value inside inner array has unexpected type)
{
  "multiValueHeaders": [
    "x-header-foo": [
      null
    ]
  ]
  
}

Summary of changes

Primary fix

  • when iterating over the header values for a given key in the multiValueHeaders object, we now explicitly confirm that the value is a string before assigning it to a variable of type std::string
- const auto& header_value = inner_it.value().get<std::string>();
+ std::string header_value;
+ if (inner_it.value().is_string()) {
+   header_value = inner_it.value().get<std::string>();
+ } else {
+   header_value = inner_it.value().dump();
+ }

related fix

  • this PR also validates that the value pointed to by a given header name in the multiValueHeaders object is an array before attempting to iterate over it
  • specifically, we reject the request if the value is of type object
  • singleton types are processed without issue, so we do not need to reject requests which contain these types
// need to validate that header_values is an array/iterable
if (!header_values.is_array()) {
  // if it's an object, we reject the request
  if (header_values.is_object()) {
    ENVOY_STREAM_LOG(debug, "invalid multi header value object", stream_filter_callbacks);
    ApiGatewayError error = {500, "500", "invalid multi header value object"};
    return ApiGatewayTransformer::format_error(*response_headers, body, error, stream_filter_callbacks);
  }

  ENVOY_STREAM_LOG(debug, "warning: using non-array value for multi header value", stream_filter_callbacks);
}

status code type checking

  • this PR also adds some additional type checking around the status field object
  • in particular, we remove an unecessary try/catch block, and get stricter about negative/float/non-integer status code values
- try {
-   status_value = json_body["statusCode"].get<uint64_t>();
- } catch (std::exception& exception){
-   ENVOY_STREAM_LOG(debug, "Error parsing statusCode: " + std::string(exception.what()), stream_filter_callbacks);
-   ApiGatewayError error = {500, "500", "Non-integer status code"};
+ if (!json_body["statusCode"].is_number_unsigned()) {
+   // add duplicate log line to not break tests for now
+   ENVOY_STREAM_LOG(debug, "cannot parse non unsigned integer status code", stream_filter_callbacks);
+   ENVOY_STREAM_LOG(debug, "received status code with value: " + json_body["statusCode"].dump(), stream_filter_callbacks);
+   ApiGatewayError error = {500, "500", "cannot parse non unsigned integer status code"};
  return ApiGatewayTransformer::format_error(*response_headers, body, error, stream_filter_callbacks);
}
+ status_value = json_body["statusCode"].get<uint64_t>();

Comment on lines -6 to -12
#include "test/mocks/server/mocks.h"
#include "test/test_common/utility.h"

#include "fmt/format.h"
#include "nlohmann/json.hpp"
#include "gmock/gmock.h"
#include "gtest/gtest.h"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed (along with the changes to the BUILD file) to improve compilation times for these tests

on my machine, this reduced the time to compile small changes to the already-cached suite from 80s to 60s

@solo-changelog-bot
Copy link

Issues linked to changelog:
solo-io/gloo#8867

@ben-taussig-solo ben-taussig-solo marked this pull request as ready for review April 18, 2024 16:10

if (expected_error_message.empty()) {
transformer.transform(response_headers, &headers, *bodyPtr, filter_callbacks_);
auto lowercase_multi_header_name = Http::LowerCaseString("test-multi-header");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would be nice to move "test-multi-header" to an external const value

ApiGatewayTransformer transformer;
NiceMock<Http::MockStreamDecoderFilterCallbacks> filter_callbacks_{};

if (expected_error_message.empty()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would prefer to use std::optional instead of empty strings as sentinel values

void test_multi_value_headers_transformation(
std::unique_ptr<Buffer::OwnedImpl> bodyPtr,
std::string expected_error_message = "Error transforming response: [json.exception.type_error.302] type must be string, but is null",
std::string expected_multi_value_header = "multi-value-0,multi-value-1,multi-value-2") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would prefer to see these as external const values

@soloio-bulldozer soloio-bulldozer bot merged commit f540a16 into main Apr 23, 2024
4 checks passed
@soloio-bulldozer soloio-bulldozer bot deleted the lambda-json-parsing branch April 23, 2024 13:57
bewebi pushed a commit that referenced this pull request May 8, 2024
* messy early commit: reduce test deps, annotate api gateway transformer, tests to describe failure modes

* safety adjustments to api_gateway_transformer.cc

* clean up some comments in api_gateway_transformer.cc

* messy update to transformer_teest.cc

* remove duplicate log line

* clean up unnecessary tests, todos

* reintroduce transform_single_and_multi_value_headers test

* remove other unintentionally removed tests

* additional test cleanup

* fix whitespace changes + address remaining comments in tests

* add changelog entry
@bewebi bewebi mentioned this pull request May 8, 2024
bewebi pushed a commit that referenced this pull request May 8, 2024
* messy early commit: reduce test deps, annotate api gateway transformer, tests to describe failure modes

* safety adjustments to api_gateway_transformer.cc

* clean up some comments in api_gateway_transformer.cc

* messy update to transformer_teest.cc

* remove duplicate log line

* clean up unnecessary tests, todos

* reintroduce transform_single_and_multi_value_headers test

* remove other unintentionally removed tests

* additional test cleanup

* fix whitespace changes + address remaining comments in tests

* add changelog entry
@bewebi bewebi mentioned this pull request May 8, 2024
bewebi pushed a commit that referenced this pull request May 8, 2024
* messy early commit: reduce test deps, annotate api gateway transformer, tests to describe failure modes

* safety adjustments to api_gateway_transformer.cc

* clean up some comments in api_gateway_transformer.cc

* messy update to transformer_teest.cc

* remove duplicate log line

* clean up unnecessary tests, todos

* reintroduce transform_single_and_multi_value_headers test

* remove other unintentionally removed tests

* additional test cleanup

* fix whitespace changes + address remaining comments in tests

* add changelog entry
@bewebi bewebi mentioned this pull request May 8, 2024
soloio-bulldozer bot pushed a commit that referenced this pull request May 8, 2024
* Fix: Lambda json parsing (#328)

* messy early commit: reduce test deps, annotate api gateway transformer, tests to describe failure modes

* safety adjustments to api_gateway_transformer.cc

* clean up some comments in api_gateway_transformer.cc

* messy update to transformer_teest.cc

* remove duplicate log line

* clean up unnecessary tests, todos

* reintroduce transform_single_and_multi_value_headers test

* remove other unintentionally removed tests

* additional test cleanup

* fix whitespace changes + address remaining comments in tests

* add changelog entry

* move changelog

---------

Co-authored-by: Ben Taussig <85883594+ben-taussig-solo@users.noreply.github.com>
soloio-bulldozer bot pushed a commit that referenced this pull request May 8, 2024
* Fix: Lambda json parsing (#328)

* messy early commit: reduce test deps, annotate api gateway transformer, tests to describe failure modes

* safety adjustments to api_gateway_transformer.cc

* clean up some comments in api_gateway_transformer.cc

* messy update to transformer_teest.cc

* remove duplicate log line

* clean up unnecessary tests, todos

* reintroduce transform_single_and_multi_value_headers test

* remove other unintentionally removed tests

* additional test cleanup

* fix whitespace changes + address remaining comments in tests

* add changelog entry

* move changelog

---------

Co-authored-by: Ben Taussig <85883594+ben-taussig-solo@users.noreply.github.com>
soloio-bulldozer bot pushed a commit that referenced this pull request May 8, 2024
* Fix: Lambda json parsing (#328)

* messy early commit: reduce test deps, annotate api gateway transformer, tests to describe failure modes

* safety adjustments to api_gateway_transformer.cc

* clean up some comments in api_gateway_transformer.cc

* messy update to transformer_teest.cc

* remove duplicate log line

* clean up unnecessary tests, todos

* reintroduce transform_single_and_multi_value_headers test

* remove other unintentionally removed tests

* additional test cleanup

* fix whitespace changes + address remaining comments in tests

* add changelog entry

* move changelog

---------

Co-authored-by: Ben Taussig <85883594+ben-taussig-solo@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants