fix: prevent retention service creating orphaned shard files #24530

gwossum · 2023-12-21T23:42:31Z

Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users.

This corrects that by requiring shards files be deleted before they can be removed from the meta store.

Closes #24529

I've read the contributing section of the project README.
Signed CLA (if not already signed).

Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. fixes: #24529

davidby-influx

This is complex enough I think I need a walk-through. Sorry that I am not fully grasping how this works.

davidby-influx · 2023-12-26T21:09:16Z

This is complex enough I think I need a walk-through. Sorry that I am not fully grasping how this works.

What I am missing is how the scenario where the retention service leaves shards on disk but deletes them from the meta-data occurs, and how these changes fix that scenario. You explained the scenario to me once, but I don't remember enough details to walk through it with the new code....

services/meta/data.go

services/retention/service.go

davidby-influx

A few comment typos, and a suggestion to simplify a little code.

services/meta/data.go

cmd/influxd/run/server.go

services/retention/service.go

…se optimizations

davidby-influx

One more change requested, an additional log message.

services/retention/service.go

davidby-influx · 2024-01-03T18:48:54Z

services/retention/service.go

+	for _, id := range s.TSDBStore.ShardIDs() {
+		if info, ok := deletedShardIDs[id]; ok {
+			delete(deletedShardIDs, id)
+			if err := s.TSDBStore.DeleteShard(id); err != nil {


One more change - what do you think about printing one more message before calling DeleteShard:

attempting deletion of shard X, database Y, retention policy Z

We have no visibility into why the retention service hangs for so long, and this might give us more clues; we can look at the size and cardinality of a shard that prints the message and then never prints a success or failure message.

davidby-influx

Yay!

Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. Backport via clean cherry-pick of #24530. fixes: #24543 (cherry picked from commit 7bd3f89)

Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. fixes: #24529 (cherry picked from commit 7bd3f89) closes #24545

…#24547) Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. fixes: #24529 (cherry picked from commit 7bd3f89) closes #24545 Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com>

…#24547) Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. fixes: #24529 (cherry picked from commit 7bd3f89) closes #24545 Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com> (cherry picked from commit 0dc48b1) closes #24546

…#24547) (#24548) Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. fixes: #24529 (cherry picked from commit 7bd3f89) closes #24545 Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com> (cherry picked from commit 0dc48b1) closes #24546

…#24547) Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. closes: #25116 (cherry picked from commit 7bd3f89) closes #24545 Co-authored-by: Geoffrey Wossum <gwossum@influxdata.com> (cherry picked from commit 0dc48b1)

…ata#24530) * fix: prevent retention service creating orphaned shard files Under certain circumstances, the retention service can fail to delete shards from the store in a timely manner. When the shard groups are pruned based on age, this leaves orphaned shard files on the disk. The retention service will then not attempt to remove the obsolete shard files because the meta store does not know about them. This can cause excessive disk space usage for some users. This corrects that by requiring shards files be deleted before they can be removed from the meta store. fixes: influxdata#24529

gwossum added 2 commits December 21, 2023 17:34

davidby-influx assigned gwossum Dec 22, 2023

davidby-influx self-requested a review December 22, 2023 00:47

davidby-influx added area/retention policies kind/bug 1.x team/edge labels Dec 22, 2023