-
Notifications
You must be signed in to change notification settings - Fork 81
Add support for C++ benchmarks. #99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mcopik
wants to merge
25
commits into
master
Choose a base branch
from
cpp_benchmarks
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
4ef9e38
[aws][system] Add C++ dependencies images
mcopik 2dc3894
[aws][system] Add C++ dependencies images
mcopik 3dd45ef
[aws] Change name style of C++ Docker images
mcopik 72b88ad
[aws] Add AWS C++ Lambda runtime as dependency
mcopik 1cdcc74
[aws] Add the basic wrapper for C++
mcopik 17c69fe
[aws] Add Docker build container for C++
mcopik 53f81f7
[benchmarks] Add sleep benchmark implementation for C++
mcopik 67560c8
[aws] Define necessary package files for C++
mcopik a9b5218
[system] Add Docker build arguments for C++ images
mcopik 3c3700c
[aws] Implement correct return from C++
mcopik 61bbffa
[aws] [system] First version of support for C++ benchmarks
mcopik 89d29e7
[aws] Add Boost UUID in handler
mcopik 5db9fc2
[aws] Support HTTP trigger for C++
mcopik c6a1a81
[aws] Minor update of C++ dependencies
mcopik dc3df69
[aws] Add key-value wrapper for C++ benchmarks
mcopik eb5d0b6
[benchmarks] Linting
mcopik 2f035fc
[system] Add logging of C++ build steps
mcopik 1c5d85c
[aws] Add Redis wrappers for C++
mcopik 9a96d72
[aws] Move Dockerfiles to a new directory
mcopik b5fbb12
[dev] Linting
mcopik a8239a1
[aws] Rename C++ utility function and apply formatting
mcopik 8e1f93a
[aws] Pin hiredis version
mcopik b7ab8e4
[aws] Fix merge error
mcopik 068adfd
[aws] Fix various bugs for C++ benchmarks, as pointed out during review
mcopik ef25201
[aws] Fix generation of cloud runtime
mcopik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
{ | ||
"timeout": 120, | ||
"memory": 128, | ||
"languages": ["python", "nodejs"], | ||
"languages": ["python", "nodejs", "cpp"], | ||
"modules": [] | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
|
||
#include <aws/core/Aws.h> | ||
#include <aws/core/utils/json/JsonSerializer.h> | ||
#include <aws/lambda-runtime/runtime.h> | ||
|
||
#include <thread> | ||
#include <iostream> | ||
|
||
Aws::Utils::Json::JsonValue function(Aws::Utils::Json::JsonView json) | ||
{ | ||
int sleep = json.GetInteger("sleep"); | ||
|
||
std::chrono::seconds timespan(sleep); | ||
std::this_thread::sleep_for(timespan); | ||
mcopik marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
//std::string res_json = "{ \"result\": " + std::to_string(sleep) + "}"; | ||
//return aws::lambda_runtime::invocation_response::success(res_json, "application/json"); | ||
Aws::Utils::Json::JsonValue val; | ||
val.WithObject("result", std::to_string(sleep)); | ||
return val; | ||
} | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
#include <chrono> | ||
#include <cstdlib> | ||
|
||
#include <aws/core/Aws.h> | ||
#include <aws/lambda-runtime/runtime.h> | ||
#include <aws/s3/S3Client.h> | ||
|
||
#include <boost/uuid/uuid.hpp> | ||
#include <boost/uuid/uuid_generators.hpp> | ||
#include <boost/uuid/uuid_io.hpp> | ||
|
||
#include "utils.hpp" | ||
mcopik marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
// Global variables that are retained across function invocations | ||
bool cold_execution = true; | ||
std::string container_id = ""; | ||
std::string cold_start_var = ""; | ||
|
||
Aws::Utils::Json::JsonValue function(Aws::Utils::Json::JsonView req); | ||
|
||
aws::lambda_runtime::invocation_response handler(aws::lambda_runtime::invocation_request const &req) | ||
{ | ||
Aws::Utils::Json::JsonValue json(req.payload); | ||
Aws::Utils::Json::JsonView json_view = json.View(); | ||
// HTTP trigger with API Gateaway sends payload as a serialized JSON | ||
// stored under key 'body' in the main JSON | ||
// The SDK trigger converts everything for us | ||
if(json_view.ValueExists("body")){ | ||
Aws::Utils::Json::JsonValue parsed_body{json_view.GetString("body")}; | ||
json = std::move(parsed_body); | ||
json_view = json.View(); | ||
} | ||
|
||
const auto begin = std::chrono::system_clock::now(); | ||
auto ret = function(json.View()); | ||
const auto end = std::chrono::system_clock::now(); | ||
|
||
Aws::Utils::Json::JsonValue body; | ||
body.WithObject("result", ret); | ||
|
||
auto b = std::chrono::duration_cast<std::chrono::microseconds>(begin.time_since_epoch()).count() / 1000.0 / 1000.0; | ||
auto e = std::chrono::duration_cast<std::chrono::microseconds>(end.time_since_epoch()).count() / 1000.0 / 1000.0; | ||
body.WithDouble("begin", b); | ||
body.WithDouble("end", e); | ||
body.WithDouble("results_time", e - b); | ||
body.WithString("request_id", req.request_id); | ||
body.WithBool("is_cold", cold_execution); | ||
body.WithString("container_id", container_id); | ||
body.WithString("cold_start_var", cold_start_var); | ||
|
||
// Switch cold execution after the first one. | ||
if(cold_execution) | ||
cold_execution = false; | ||
|
||
Aws::Utils::Json::JsonValue final_result; | ||
final_result.WithObject("body", body); | ||
return aws::lambda_runtime::invocation_response::success(final_result.View().WriteReadable(), "application/json"); | ||
} | ||
|
||
int main() | ||
{ | ||
Aws::SDKOptions options; | ||
Aws::InitAPI(options); | ||
|
||
const char * cold_var = std::getenv("cold_start"); | ||
if(cold_var) | ||
cold_start_var = cold_var; | ||
container_id = boost::uuids::to_string(boost::uuids::random_generator()()); | ||
|
||
aws::lambda_runtime::run_handler(handler); | ||
|
||
Aws::ShutdownAPI(options); | ||
return 0; | ||
} | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
|
||
#include <chrono> | ||
#include <thread> | ||
|
||
#include <aws/core/auth/AWSCredentialsProvider.h> | ||
#include <aws/core/client/ClientConfiguration.h> | ||
#include <aws/dynamodb/model/AttributeDefinition.h> | ||
#include <aws/dynamodb/model/GetItemRequest.h> | ||
#include <aws/dynamodb/model/PutItemRequest.h> | ||
#include <aws/dynamodb/model/PutItemResult.h> | ||
|
||
#include <boost/interprocess/streams/bufferstream.hpp> | ||
|
||
#include "key-value.hpp" | ||
#include "utils.hpp" | ||
|
||
KeyValue::KeyValue() { | ||
Aws::Client::ClientConfiguration config; | ||
// config.region = "eu-central-1"; | ||
config.caFile = "/etc/pki/tls/certs/ca-bundle.crt"; | ||
|
||
char const TAG[] = "LAMBDA_ALLOC"; | ||
auto credentialsProvider = | ||
Aws::MakeShared<Aws::Auth::EnvironmentAWSCredentialsProvider>(TAG); | ||
_client.reset(new Aws::DynamoDB::DynamoDBClient(credentialsProvider, config)); | ||
} | ||
|
||
uint64_t KeyValue::download_file(Aws::String const &table, | ||
Aws::String const &key, int &required_retries, | ||
double &read_units, bool with_backoff) { | ||
Aws::DynamoDB::Model::GetItemRequest req; | ||
|
||
// Set up the request | ||
req.SetTableName(table); | ||
req.SetReturnConsumedCapacity( | ||
Aws::DynamoDB::Model::ReturnConsumedCapacity::TOTAL); | ||
Aws::DynamoDB::Model::AttributeValue hashKey; | ||
hashKey.SetS(key); | ||
req.AddKey("key", hashKey); | ||
|
||
auto bef = timeSinceEpochMicrosec(); | ||
int retries = 0; | ||
const int MAX_RETRIES = 1500; | ||
mcopik marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
while (retries < MAX_RETRIES) { | ||
auto get_result = _client->GetItem(req); | ||
if (get_result.IsSuccess()) { | ||
|
||
// Reference the retrieved fields/values | ||
auto result = get_result.GetResult(); | ||
const Aws::Map<Aws::String, Aws::DynamoDB::Model::AttributeValue> &item = | ||
result.GetItem(); | ||
if (item.size() > 0) { | ||
uint64_t finishedTime = timeSinceEpochMicrosec(); | ||
|
||
required_retries = retries; | ||
// GetReadCapacityUnits returns 0? | ||
read_units = result.GetConsumedCapacity().GetCapacityUnits(); | ||
|
||
return finishedTime - bef; | ||
} | ||
|
||
} else { | ||
retries += 1; | ||
if (with_backoff) { | ||
int sleep_time = retries; | ||
if (retries > 100) { | ||
sleep_time = retries * 2; | ||
} | ||
std::this_thread::sleep_for(std::chrono::milliseconds(sleep_time)); | ||
} | ||
} | ||
} | ||
return 0; | ||
} | ||
|
||
uint64_t KeyValue::upload_file(Aws::String const &table, Aws::String const &key, | ||
double &write_units, int size, | ||
unsigned char *pBuf) { | ||
Aws::Utils::ByteBuffer buf(pBuf, size); | ||
|
||
Aws::DynamoDB::Model::PutItemRequest req; | ||
req.SetTableName(table); | ||
req.SetReturnConsumedCapacity( | ||
Aws::DynamoDB::Model::ReturnConsumedCapacity::TOTAL); | ||
|
||
Aws::DynamoDB::Model::AttributeValue av; | ||
av.SetB(buf); | ||
req.AddItem("data", av); | ||
av.SetS(key); | ||
req.AddItem("key", av); | ||
|
||
uint64_t bef = timeSinceEpochMicrosec(); | ||
const Aws::DynamoDB::Model::PutItemOutcome put_result = _client->PutItem(req); | ||
if (!put_result.IsSuccess()) { | ||
std::cout << put_result.GetError().GetMessage() << std::endl; | ||
return 1; | ||
} | ||
auto result = put_result.GetResult(); | ||
// GetWriteCapacityUnits returns 0? | ||
write_units = result.GetConsumedCapacity().GetCapacityUnits(); | ||
uint64_t finishedTime = timeSinceEpochMicrosec(); | ||
|
||
return finishedTime - bef; | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
|
||
#include <cstdint> | ||
#include <string> | ||
#include <initializer_list> | ||
#include <memory> | ||
|
||
#include <aws/core/utils/memory/stl/AWSString.h> | ||
#include <aws/dynamodb/DynamoDBClient.h> | ||
|
||
class KeyValue | ||
{ | ||
// non-copyable, non-movable | ||
std::shared_ptr<Aws::DynamoDB::DynamoDBClient> _client; | ||
public: | ||
|
||
KeyValue(); | ||
|
||
uint64_t download_file(Aws::String const &bucket, | ||
Aws::String const &key, | ||
int& required_retries, | ||
double& read_units, | ||
bool with_backoff = false); | ||
|
||
uint64_t upload_file(Aws::String const &bucket, | ||
Aws::String const &key, | ||
double& write_units, | ||
int size, | ||
unsigned char* pBuf); | ||
|
||
}; | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
|
||
#include <chrono> | ||
#include <iostream> | ||
#include <string> | ||
#include <thread> | ||
|
||
#include <aws/core/Aws.h> | ||
|
||
#include "redis.hpp" | ||
#include "utils.hpp" | ||
|
||
Redis::Redis(std::string redis_hostname, int redis_port) { | ||
_context = redisConnect(redis_hostname.c_str(), redis_port); | ||
if (_context == nullptr || _context->err) { | ||
if (_context) { | ||
std::cerr << "Redis Error: " << _context->errstr << '\n'; | ||
} else { | ||
std::cerr << "Can't allocate redis context\n"; | ||
} | ||
} | ||
} | ||
|
||
bool Redis::is_initialized() { return _context != nullptr; } | ||
|
||
Redis::~Redis() { redisFree(_context); } | ||
|
||
uint64_t Redis::download_file(Aws::String const &key, int &required_retries, | ||
bool with_backoff) { | ||
std::string comm = "GET " + key; | ||
|
||
auto bef = timeSinceEpochMicrosec(); | ||
int retries = 0; | ||
const int MAX_RETRIES = 1500; | ||
|
||
while (retries < MAX_RETRIES) { | ||
|
||
redisReply *reply = (redisReply *)redisCommand(_context, comm.c_str()); | ||
|
||
if (reply->type == REDIS_REPLY_NIL || reply->type == REDIS_REPLY_ERROR) { | ||
|
||
retries += 1; | ||
if (with_backoff) { | ||
int sleep_time = retries; | ||
if (retries > 100) { | ||
sleep_time = retries * 2; | ||
} | ||
std::this_thread::sleep_for(std::chrono::milliseconds(sleep_time)); | ||
} | ||
|
||
} else { | ||
|
||
uint64_t finishedTime = timeSinceEpochMicrosec(); | ||
required_retries = retries; | ||
|
||
freeReplyObject(reply); | ||
return finishedTime - bef; | ||
} | ||
freeReplyObject(reply); | ||
} | ||
return 0; | ||
} | ||
|
||
uint64_t Redis::upload_file(Aws::String const &key, int size, char *pBuf) { | ||
std::string comm = "SET " + key + " %b"; | ||
|
||
uint64_t bef = timeSinceEpochMicrosec(); | ||
redisReply *reply = | ||
(redisReply *)redisCommand(_context, comm.c_str(), pBuf, size); | ||
uint64_t finishedTime = timeSinceEpochMicrosec(); | ||
|
||
if (reply->type == REDIS_REPLY_NIL || reply->type == REDIS_REPLY_ERROR) { | ||
std::cerr << "Failed to write in Redis!" << std::endl; | ||
abort(); | ||
} | ||
mcopik marked this conversation as resolved.
Show resolved
Hide resolved
|
||
freeReplyObject(reply); | ||
|
||
return finishedTime - bef; | ||
} | ||
|
||
uint64_t Redis::delete_file(std::string const &key) { | ||
std::string comm = "DEL " + key; | ||
|
||
uint64_t bef = timeSinceEpochMicrosec(); | ||
redisReply *reply = (redisReply *)redisCommand(_context, comm.c_str()); | ||
uint64_t finishedTime = timeSinceEpochMicrosec(); | ||
|
||
if (reply->type == REDIS_REPLY_NIL || reply->type == REDIS_REPLY_ERROR) { | ||
std::cerr << "Couldn't delete the key!" << '\n'; | ||
abort(); | ||
} | ||
freeReplyObject(reply); | ||
|
||
return finishedTime - bef; | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
|
||
#include <cstdint> | ||
#include <string> | ||
|
||
#include <aws/core/utils/memory/stl/AWSString.h> | ||
|
||
#include <hiredis/hiredis.h> | ||
|
||
class Redis | ||
{ | ||
redisContext* _context; | ||
public: | ||
|
||
Redis(std::string redis_hostname, int redis_port); | ||
~Redis(); | ||
|
||
bool is_initialized(); | ||
|
||
uint64_t download_file(Aws::String const &key, int &required_retries, bool with_backoff); | ||
|
||
uint64_t upload_file(Aws::String const &key, int size, char* pBuf); | ||
|
||
uint64_t delete_file(std::string const &key); | ||
|
||
}; | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.