Closed
Description
Request Type
Bug
Work Environment
Question | Answer |
---|---|
OS version (server) | Ubuntu |
OS version (client) | XP, Seven, 10, Ubuntu, ... |
Virtualized Env. | True |
Dedicated RAM | 8 GB |
vCPU | 2 |
TheHive version / git hash | thehiveproject/thehive4:4.1.16-1 (image commit: 7b59e21f2442d077928d896dddcaf72fde657d61) |
Package Type | Docker, |
Database | Cassandra |
Index type | Elasticsearch |
Attachments storage | S3 |
Browser type & version | If applicable |
Problem Description
When migrating data from The Hive 3.5.1 to 4.1.16 and using AWS S3 as the storage backend for the 4.1.16 cluster, file uploads fail with a 400 Bad Request
due to missing uploadId
.
Steps to Reproduce
- Setup The Hive 4 with S3 storage backend. Relevant config section:
storage {
provider: s3
s3 {
bucket = "bucket-name"
readTimeout = 1 minute
writeTimeout = 1 minute
chunkSize = 1 MB
endpoint = "https://s3.eu-central-1.amazonaws.com"
region = "eu-central-1"
accessKey = ""
secretKey = ""
}
}
alpakka.s3.aws.credentials.provider = default
alpakka.s3.access-style = virtual
NB! Using IAM role for auth, hence setting alpakka credentials provider to default
-
Start migration with:
/opt/thehive/bin/migrate -d --output /etc/thehive/application.conf --main-organisation OrgName --input /etc/thehive/hive3.conf
-
Migration starts warning on each S3 upload with the following:
2022-01-09 19:01:22,218 [WARN] from org.thp.thehive.migration.th4.Output in TheHiveMigration-akka.actor.default-dispatcher-16 [|55ea1a97] Unable to set avatar to user xxx.xxx@xxx.com: akka.stream.alpakka.s3.FailedUpload: Upload part 1 request failed. Response header: (HttpResponse(400 Bad Request,List(x-amz-request-id: 1PWGZ6FZ51TC6PQE, x-amz-id-2: dgDz0+nxd6CkG3jtwgeoOhxrm2YlUhUhp3PQLzaS7NhsGvkcnF/Ps7H4zNZ22XQP7bdNEu1sF3g=, Date: Sun, 09 Jan 2022 19:01:21 GMT, Server: AmazonS3, Connection: close),HttpEntity.Chunked(application/xml),HttpProtocol(HTTP/1.1))), response body: (<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>This operation does not accept partNumber without uploadId</Message><ArgumentName>partNumber</ArgumentName><ArgumentValue>partNumber</ArgumentValue><RequestId>1PWGZ6FZ51TC6PQE</RequestId><HostId>xxx</HostId></Error>).
- No files are uploaded to S3 bucket
Possible Solutions
- The Hive is currently using alpakka 2.0.2, but It seems that alpakka 3.0.4 introduces quite substantial changes to the AWS S3 library, which can possibly remedy the situation.
- Since the core problem seems to be a missing
uploadId
value for the MultiPartUpload and there seem to be references to it in alpakkas 2.0.2 version here then possibly setting theuploadId
when calling the alpakka MultiPartUpload function could also fix it.