compact/downsampling: Build chunks directly in object storage (without using disk)

[bwplotka]

It's totally doable now. Same with downsampling.

[pracucci]

Do you mean read source block chunks from object store, merge them on the fly and stream the output to the compacted block directly to the object store without touching disk?

[bwplotka]

Correct. Or at least touching some constant amount of disk. Exploring this right now.

[bwplotka]

The most complex part of it is that we need to extend API with PartUpload to do such streaming: https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html

Or we can add even use PartCopy to avoid touching those bytes directly.

[bwplotka]

Another limit for S3 is Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.

For GCS we could use just https://cloud.google.com/storage/docs/resumable-uploads because https://cloud.google.com/storage/docs/json_api/v1/how-tos/multipart-upload is used for something else (metadata upload). On top of that: https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite

[bwplotka]

Azure is very flexible for this: https://docs.microsoft.com/en-us/rest/api/storageservices/#blob-service

Damn it's a lot of work 😢

But the gain is trivial: We don't need disk (or a lot of disks) to write chunks into chunk files. The problem of the disk is not that urging though.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[GiedriusS]

Not sure if I agree with the "trivial" part. In bigger deployments, the size of produced blocks can easily go into the range of hundreds of gigabytes. It would be very nice to have this so hence reopening.

[GiedriusS]

I actually had a troll-ish idea, it might be already feasible. What if we mounted, for example, an S3 bucket via s3fs (https://github.com/s3fs-fuse/s3fs-fuse) and then used the filesystem storage type for Thanos Compactor? In the compaction case, we download whole files so it's not an issue that Thanos is unaware of the underlying storage, right? Any thoughts @pracucci @bwplotka ?

[bwplotka]

That might work for a start! (: The problem with this approach is local disk data caching at the end might either end up downloading things on disk or make an additional number of calls unnecessarily. What we want is to have full control over those things.

I agree it's not trivial, but it's also not super difficult. Definitely high on priority list. Maybe good project for LFX summer? 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[someshkoli]

Was looking for solutions to pull -> transform-> upload very large file to and from obj storage (S3 in my case), thought ther would be some implementations here on thanos :P looks like we're yet to implement this.
Used part upload for my purpose and it works fine, will try to onboard the same here. Will do a poc for S3 and then we can move forward for other services too.

[GiedriusS]

Doing this for compaction is a bit trickier since Prometheus heavily relies on mmap. Downsampling is totally doable, though, since everything is under an interface.

[vjabrayilov]

Hey, I would like to resolve this issue and enhance compaction/downsampling. I have written a proposal for GSoC. It would be kind of you if you review and give feedback. @bwplotka @yeya24 @matej-g
https://drive.google.com/file/d/1oZd3ENSZ7v2hONNf4pDL3yXr_0eHcwsV/view?usp=sharing

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[yeya24]

Still valid

[oronsh]

@yeya24 @bwplotka @GiedriusS Doesn't it mean that for 2 blocks having 1M series each and all series are the same, you'd have to have 2M requests for S3 copying series A from block A and series A from block B chunks to the new block C chunk file?

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.