Mock large message (>1kb) handling in create blob v2 action

[xernaj]

Hi,

Has anyone run into an issue with mocking large messages (>1kb) in a create blob v2 action? (macos)

status='Failed', statusCode='InvalidProtocolResponse', error='{"code":"InvalidProtocolResponse","message":"The response to partial content upload initiating request is not valid. The response to initiating partial content upload request must contain a valid location header."}

If I mock the "Location" header then this error appears:

{"code":"InvalidProtocolResponse","message":"The response to partial content upload request is not valid. The location header value returned in the response 'https%3A%2F%2Flocalhost%3A7075%2Fchunk' must be a well formed absolute URI not referencing local host or UNC path."}

It appears that chunking can't be mocked so I've started on something that will disable chunking. Hopefully this is the right path?

[mark-abrams]

Hi @xernaj , thanks for creating this issue. I've created a simple workflow that receives a JSON document via a HTTP trigger and uploads the document as a blob to Azure Storage, using the uploadBlob built-in connector:

I've also written a test to test the workflow, and validate that the content body of the HTTP trigger matches the content in the request that is created for the "Upload Blob" action:

The workflow is successful when run with the test, the workflow completes successfully and the test is successful. I've tested this with both Windows and Ubuntu - I don't have access to a macOS laptop.

Is your scenario different to the one that I've described above?

[mark-abrams]

I forgot to add that I'm using a message that is 10Kb in size when saved to blob storage.

[xernaj]

Hi Mark, I'll try reproduce that workflow and see what happens on my Mac (Intel).

[xernaj]

Hi @mark-abrams ,

I've created a sample workflow:

• recurrence trigger
• http get a >1kb json file
• http post body from get output body

I noticed it also appears in managed create blob connector (not in upload built-in connector).

Notice the "content transfer" flag which appears in JSON as:

"runtimeConfiguration": {
    "contentTransfer": {
        "transferMode": "Chunked"
    }
}

Error when running test:
2023-07-27T12:12:18.181Z] Workflow action ends. flowName='chunked-transfer-workflow', actionName='POST', flowId='7e5d96553e6f46359b1f8dd1ee16f823', flowSequenceId='08585111469504939287', flowRunSequenceId='08585111469484371867616577762CU00', correlationId='799fd51a-0c33-4b0d-af47-0d4809045356', status='Failed', statusCode='InvalidProtocolResponse', error='{"code":"InvalidProtocolResponse","message":"The response to partial content upload initiating request is not valid. The response to initiating partial content upload request must contain a valid location header."}', durationInMilliseconds='90', inputsContentSize='1399', outputsContentSize='-1', extensionVersion='1.28.4.0', siteName='UNDEFINED_SITE_NAME', slotName='', actionTrackingId='2c1c442f-e211-4da9-b0b9-b63a26c5f19a', clientTrackingId='08585111469484371867616577762CU00', properties='{"$schema":"2016-06-01","startTime":"2023-07-27T12:12:18.090379Z","endTime":"2023-07-27T12:12:18.181308Z","status":"Failed","code":"InvalidProtocolResponse","executionClusterType":"Classic","resource":{"workflowId":"7e5d96553e6f46359b1f8dd1ee16f823","workflowName":"chunked-transfer-workflow","runId":"08585111469484371867616577762CU00","actionName":"POST"},"correlation":{"actionTrackingId":"2c1c442f-e211-4da9-b0b9-b63a26c5f19a","clientTrackingId":"08585111469484371867616577762CU00"},"error":{"code":"InvalidProtocolResponse","message":"The response to partial content upload initiating request is not valid. The response to initiating partial content upload request must contain a valid location header."},"api":{},"isV2Threshold":false}', actionType='Http', sequencerType='Linear', flowScaleUnit='cu00', platformOptions='RunDistributionAcrossPartitions, RepetitionsDistributionAcrossSequencers, RunIdTruncationForJobSequencerIdDisabled, RepetitionPreaggregationEnabled', retryHistory='', failureCause='', overrideUsageConfigurationName='', hostName='localhost', activityId='3169a1fa-1958-46cf-8214-96a70005bea6'.

I've created a fork in order to create a sample workflow and test:
https://github.com/xernaj/TestingFramework/blob/sample-chunked-transfer/LogicAppUnit.Samples.LogicApps/chunked-transfer-workflow/workflow.json
https://github.com/xernaj/TestingFramework/blob/sample-chunked-transfer/LogicAppUnit.Samples.LogicApps.Tests/ChunkedTransferWorkflowTest/ChunkedTransferWorkflowTest.cs

Please let me know if you're seeing the same issue in windows or ubuntu. If so, I can create a PR to remove the contenttransfer block.

[mark-abrams]

Hi @xernaj , now I understand the issue and I can re-create it myself. Basically you are enabling the chunking upload protocol in the HTTP action called POST and the testing framework is not implementing the "server-side" requirements for the chunking protocol to work.

As a workaround it is possible to implement the server-side requirements in a test case. This example uses a mix of the Fluent API (new in version 1.7.0) and the delegate mock function, since there is some functionality that is needed that cannot be implemented in the Fluent API.

Firstly, set up some variables in the test case to manage state of the chunking process:

Then configure a Request Matcher and Response Builder to handle the initial POST request in the chunking protocol:

Then configure a delegate mock to handle the PATCH requests that contain the chunks of data:

Once this is done, you can easily assert the contains of the stream, i.e. the data that was sent by the POST action in the workflow:

I realise that this is a bit fiddly to set up, but at least this is a workaround that works.

[mark-abrams]

There are a few ways that this could be improved:

1. Change the framework so that the runtimeConfiguration.contentTransfer setting for all HTTP actions are removed from the workflow definition before the test is run. This appraoch would be similar to how the retry policies for HTTP actions are updated to none. Might want to enable/disable this feature via a test configuration setting.
2. Change the Fluent API processing so that the "server-side" requirements for the chunking protocol are implemented internally and not visible to the test author. Maybe this additional logic could be enabled using a new EnableChunking() method in the fluent API.

Option 1 is the easiest to implement, but I'm very careful about making changes to the workflow definition. Every change to the definition moves the workflow being tested away from the workflow that would run in a production environment. This is why option 2 might be better - there would be no change required to the workflow definition and the testing framework would natively handle the chunking protocol, in a similar way to the workaround I have shown.

Also need to think if there are any test scenarios where the enabling of chunking for a HTTP action would be required. I can't think of any currently, since the chunking in the workflow is "out of the box" Logic App functionality and therefore (probably) would not be part of any test scenario.

[xernaj]

Thanks Mark, I'll give that a go. Last time I tried to mock the chunking prior to v1.70, the function app runtime complained about it not allowing chunking when calling localhost or a non FQDN - hopefully this works.

Hi @xernaj , now I understand the issue and I can re-create it myself. Basically you are enabling the chunking upload protocol in the HTTP action called POST and the testing framework is not implementing the "server-side" requirements for the chunking protocol to work.

As a workaround it is possible to implement the server-side requirements in a test case. This example uses a mix of the Fluent API (new in version 1.7.0) and the delegate mock function, since there is some functionality that is needed that cannot be implemented in the Fluent API.

I realise that this is a bit fiddly to set up, but at least this is a workaround that works.

[xernaj]

Hi @mark-abrams ,

Thanks for your effort with v1.70 fluent api - (it looks nice and clean) and looking at the chunking issue. I couldn't get your workaround to succeed on my intel mac, with func core tools "4.0.5198". On macos, it seems that the function app runtime doesn't support localhost in the Location header.

"{"code":"InvalidProtocolResponse","message":"The response to partial content upload request is not valid. The location header value returned in the response 'http://localhost:7075/api/v1.1/287124d3-a604-4b35-97fa-2cc2d7332772' must be a well formed absolute URI not referencing local host or UNC path."}'"

Here's the repo and sample test https://github.com/xernaj/TestingFramework/blob/b721c903ba5ce864e5817e31eb9d689576813f23/LogicAppUnit.Samples.LogicApps.Tests/ChunkedTransferWorkflowTest/ChunkedTransferWorkflowTest.cs#L40

I think I will look at implementing your Option 1 (user switchable disabling of chunking by modifying workflow.json) when/if time allows. My only workaround is to disable chunking - it appears to be auto enabled for HTTP and managed connector blob actions so there's a lot of workflows I may need to update to make them unit testable.

Cheers

[mark-abrams]

Hi @xernaj , I can re-produce your problem on both Ubuntu and MacOS. It seems that the URL in the Location header cannot reference localhost, or even 127.0.01. It needs to use a DNS name, even if this is the name of the host host server. I tried changing the code I shared with you earlier to use Environment.MachineName and this didn't work either - same error.

But if I change the core LogicAppUnit code so that the mock HTTP Server listens on a URL created using Environment.MachineName and not localhost it works for Windows and Ubuntu, but not for macOS! Take a look at this action workflow build where the tests are successful for Windows and Ubuntu, but failed on macOS with this error:

System.AggregateException: One or more errors occurred. (nodename nor servname provided, or not known (mac-1692870849291:7071)) ---> System.Net.Http.HttpRequestException: nodename nor servname provided, or not known (mac-1692870849291:7071) ---> System.Net.Sockets.SocketException: nodename nor servname provided, or not known

I'm not sure why this is failing on macOS. I was hoping to add support to the testing framework and the fluent API so that it automatically supports chunking in the mock HTTP server, but I'm not sure how to get this working with macOS if neither localhost nor Environment.MachineName work in the URL for the Location header. The only other option is to pre-process the workflow definition file and remove all of the chunking options for HTTP actions, but I was hoping to avoid this. If I can't think of a better solution, I will implement this in the next version (1.8.0).

[xernaj]

Thanks for looking further into it @mark-abrams . There seems to be some logic-app/func-app runtime implementation differences between the OSes :)

[mark-abrams]

Version 1.8.0 includes a new feature to remove the chunking options for HTTP actions in a workflow definition. This feature is enabled by default and can be disabled using the test configuration JSON file.