Removing snapshot is very request-intensive

[bobrik]

Looking at logs, it seems that number of requests to figure out what could be deleted and what not is calculated as number of shards x number of snapshots. In our case with daily indices for a year of data and 37 snapshots this is roughly 75000 requests. In another month it would be 150k.

If you start making daily shapshots with 2000 shards and 5 new shards every day, at the end of the year you'll need 214634600 (214 million!) requests to remove single snapshot.

This takes a lot of time to finish. Maybe there could be some optimization with denormalization to speed things up a bit.

I realize that removing a snapshot is not that frequent operation :)

[bobrik]

$8.5 to remove single snapshot with current pricing of aws :)

[tlrx]

wow, something is not going well here. Thanks for reporting. Can you please tell us which version of ES/Cloud AWS you are using?

Thanks

[rasathus]

Its interesting you should mention that, i'm in the middle of investigating a spike in requests i saw this month on a small cluster we had setup for experiments. We had some major issues across a number of indexes, which we managed to recover and rebuild. But looking at the usage, there is a major spike for the next four days or so.

Ill have a look through and see if we've still got some logs.

[bobrik]

@tlrx I'm using es 1.4.1 and aws 2.4.1, both upgraded from 1.3.x branch today.

[bobrik]

I think we could use single file for snapshots of specific shard, then delete will not depend on snapshot count. This is also backwards compatible, if compound file exists it should be used, but if not, it should be created from existing per-snapshot files.

Maybe this could be extended to indices instead of shards, since it's not possible to snapshot or restore single shard anyway.

[tlrx]

@rasathus Thanks for the info. The version 2.4.1 of the plugin uses the AWS Multipart API to upload files larger than the buffer_size setting, which is set to 5mb by default. That is, a 50mb file will be uploaded into 10 + 2 requests whereas it was uploaded by a single request before. I think this explains why you see an increased number of requests. You can increase this buffer_size (see doc - in repository setting) according to the size of your indices and the RAM of your nodes.

Note that AWS encourage users to use Multipart Upload for objects greater than 100 MB (see http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html).

[tlrx]

@bobrik Do you snapshot all your indices every day?

[bobrik]

@tlrx yep, elastic/curator#174

[tlrx]

Ok, so 2000+ shards are snapshoted every day and snapshot older than X days is deleted every day?

[bobrik]

No snapshots are deleted

[imotov]

@bobrik that strategy will lead to progressively longer and longer snapshots, which will become an issue even before the price. Because snapshots are incremental we need to know what parts are already stored in the repository and what parts are not. In order to do that we need to read all snapshots every time a new snapshot is created or old snapshot is deleted. As you astutely noticed that's a lot of reads. So, the best strategy to avoid this issue is by moving to a new repository periodically or by pruning old snapshots (keeping old snapshots at less frequent interval).

[bobrik]

@imotov what about keeping one file with data for all snapshots of a given shard? That will remove an issue with the number of requests. It will be always O(shards).

[matthughes]

I have this issue as well even though I'm following imotov's advice.

Every hour, I snapshot to my 'hourly-repo' and delete anything older than 24 hours.
Every day, I snapshot to my 'daily-repo' and delete anything older than 1 day.

However, just deleting ONE snapshot has taken more than ten minutes so far -- still waiting for it to complete. It takes longer to delete a snapshot than to take one.

I'm running a 3-node cluster with about 1900 shards, 200 indices, and 4.5 gigs of data, a pretty modest setup from what I've seen. However, it is expected to grow much larger than that, so I'm a little concerned about these snapshot times.

I'm on ES 1.3.5. Are things supposed to be better with 1.4.x?

@imotov If you moved to a new repository periodically, you could hit a huge initial snapshot. My first snapshot took 40 minutes with a not-that-large 4-gig cluster.

[matthughes]

So I did a bit more testing. For a ~100 indexes, ~500 shard system, I took 24 snapshots in a row of completely unchanging, optimized indexes. Besides the initial snapshot, snapshot times went from 38 seconds to 2 minutes.

Removal of a single snapshot took 8 minutes. Is that expected?

[imotov]

@matthughes that doesn't sound right. Thanks for the test. We will investigate.