http server: missing support for "Expect: 100-continue"

[nyh]

The HTTP 1.1 RFC has a feature "Expect: 100-continue", described in this section: https://tools.ietf.org/html/rfc7231#section-5.1.1
We are missing support for this feature.

A client which uses this feature expects a "100 continue" response before sending the request body, and if it doesn't receive it, may not send the request body at all, or delay the sending considerably. In the Scylla project which uses Seastar' HTTP server this caused a certain client to add a one-second latency to each request - see scylladb/scylladb#6844.

Tagging this issue as a "bug" because the RFC says the server MUST support this feature. It is not optional (although, granted, most clients will never use it).

[psarna]

@wmitros I'll assign you to this issue as a first Seastar HTTP task.
Below are some details on how to quite easily reproduce the problem with Scylla's alternator, loosely based on a corresponding Scylla issue: scylladb/scylladb#6844.

1. Clone the https://github.com/awsdocs/aws-doc-sdk-examples/ repo
2. Go to cpp DynamoDB examples directory -> cpp/example_code/dynamodb
3. Compile.
   • It depends on AWS CPP sdk, which you can find here: https://github.com/aws/aws-sdk-cpp . Build with -DBUILD_ONLY="dynamodb" cmake flag or it will take one billion hours to compile. Then, install with sudo make install.
   • aws-doc-sdk-examples/cpp also did not compile for me cleanly, it only started working after I explicitly asked for dynamic libraries instead of static ones: cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/usr/local/lib64
4. Edit the examples so that they try to contact a local server instead of AWS. It's generally done by adding a single line in the source files:

--- a/cpp/example_code/dynamodb/get_item.cpp
+++ b/cpp/example_code/dynamodb/get_item.cpp
@@ -75,6 +75,7 @@ int main(int argc, char** argv)
 
         // snippet-start:[dynamodb.cpp.get_item.code]
         Aws::Client::ClientConfiguration clientConfig;
+        clientConfig.endpointOverride = "http://localhost:8000";
         Aws::DynamoDB::DynamoDBClient dynamoClient(clientConfig);
         Aws::DynamoDB::Model::GetItemRequest req;

5. Run a local Scylla instance with alternator enabled. I can help you set it up, you can do that by hand or with Scylla CCM (https://github.com/scylladb/scylla/wiki/Using-CCM). Just let me know when you reach this bullet point and I can help you set it up. Alternatively, just ask somebody onsite.
6. Run the examples - create_table is enough, you can follow it with put_item and get_item to see that alternator works fine.
7. Observe that there indeed is a 1s lag when running these examples. It most likely comes from lack of 100-continue handling mentioned in Can Alternator correctly handle HTTP requests that include the "Expect: 100-continue" HTTP request header? scylladb#6844.
8. Optionally (and recommended), observe with wireshark that 100-continue header is indeed sent to localhost:8000.

Now that we have a testing environment which resembles a production workload, all it takes is to implement handling Expect: 100-continue header so that it starts working :) We should also have a unit test for that bug in order to avoid future regressions.