Skip to content

[Bug] Unable to use AWS S3 as storage backend  #2316

Closed
@andres-tw

Description

@andres-tw

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu
OS version (client) XP, Seven, 10, Ubuntu, ...
Virtualized Env. True
Dedicated RAM 8 GB
vCPU 2
TheHive version / git hash thehiveproject/thehive4:4.1.16-1 (image commit: 7b59e21f2442d077928d896dddcaf72fde657d61)
Package Type Docker,
Database Cassandra
Index type Elasticsearch
Attachments storage S3
Browser type & version If applicable

Problem Description

When migrating data from The Hive 3.5.1 to 4.1.16 and using AWS S3 as the storage backend for the 4.1.16 cluster, file uploads fail with a 400 Bad Request due to missing uploadId.

Steps to Reproduce

  1. Setup The Hive 4 with S3 storage backend. Relevant config section:
storage {
  provider: s3
  s3 {
    bucket = "bucket-name"
    readTimeout = 1 minute
    writeTimeout = 1 minute
    chunkSize = 1 MB
    endpoint = "https://s3.eu-central-1.amazonaws.com"
    region = "eu-central-1"
    accessKey = ""
    secretKey = ""
  }
}
alpakka.s3.aws.credentials.provider = default
alpakka.s3.access-style = virtual

NB! Using IAM role for auth, hence setting alpakka credentials provider to default

  1. Start migration with: /opt/thehive/bin/migrate -d --output /etc/thehive/application.conf --main-organisation OrgName --input /etc/thehive/hive3.conf

  2. Migration starts warning on each S3 upload with the following:

2022-01-09 19:01:22,218 [WARN] from org.thp.thehive.migration.th4.Output in TheHiveMigration-akka.actor.default-dispatcher-16 [|55ea1a97] Unable to set avatar to user xxx.xxx@xxx.com: akka.stream.alpakka.s3.FailedUpload: Upload part 1 request failed. Response header: (HttpResponse(400 Bad Request,List(x-amz-request-id: 1PWGZ6FZ51TC6PQE, x-amz-id-2: dgDz0+nxd6CkG3jtwgeoOhxrm2YlUhUhp3PQLzaS7NhsGvkcnF/Ps7H4zNZ22XQP7bdNEu1sF3g=, Date: Sun, 09 Jan 2022 19:01:21 GMT, Server: AmazonS3, Connection: close),HttpEntity.Chunked(application/xml),HttpProtocol(HTTP/1.1))), response body: (<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>This operation does not accept partNumber without uploadId</Message><ArgumentName>partNumber</ArgumentName><ArgumentValue>partNumber</ArgumentValue><RequestId>1PWGZ6FZ51TC6PQE</RequestId><HostId>xxx</HostId></Error>).
  1. No files are uploaded to S3 bucket

Possible Solutions

  • The Hive is currently using alpakka 2.0.2, but It seems that alpakka 3.0.4 introduces quite substantial changes to the AWS S3 library, which can possibly remedy the situation.
  • Since the core problem seems to be a missing uploadId value for the MultiPartUpload and there seem to be references to it in alpakkas 2.0.2 version here then possibly setting the uploadId when calling the alpakka MultiPartUpload function could also fix it.

Metadata

Metadata

Assignees

Labels

TheHive4TheHive4 related issuesbug

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions