Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prune old Time Machine backups from the back end #22033

Closed
yolabingo opened this issue Apr 18, 2022 · 7 comments · Fixed by #28586 or #28665
Closed

Prune old Time Machine backups from the back end #22033

yolabingo opened this issue Apr 18, 2022 · 7 comments · Fixed by #28586 or #28665
Assignees
@dsilvam
Labels
Merged OKR : Customer Support Owned by Scott OKR : Technical User Experience Owned by Freddy Priority : 3 Average QA : Approved QA : Passed Internal Release : 24.05.31 Bug Fixing Team : Falcon Type : Enhancement

Comments

@yolabingo
Copy link
Contributor

yolabingo commented Apr 18, 2022 •
edited
Loading

Is your feature request related to a problem? Please describe.
Time Machine backup files are saved indefinitely. Over time they can consume a substantial amount of disk space.

I believe the only way to remove these files from the dotCMS backend is to delete a Time Machine backup job, which will delete all backups for that job.

Otherwise, pruning old Time Machine files requires direct access to the file system.

Describe the solution you'd like
A couple suggestions:

  1. manual BE delete - allow BE admins to "delete Backups older than dd-mm-yyyy" or "delete all but the previous N backups"
  2. automated - specify "number of backups to save" on the Time Machine job
@swicken-dotcms swicken-dotcms added the LTS: Discuss To be discussed between Support & R&D label Apr 19, 2022
@wezell
Copy link
Contributor

wezell commented May 3, 2022

We should have a config option called:
PRUNE_TIMEMACHINE_OLDER_THAN_DAYS

with a default of 90. We should delete timemachine snapshots older than the configed number of days.

@swicken-dotcms swicken-dotcms removed the LTS: Discuss To be discussed between Support & R&D label May 3, 2022
@stale
Copy link

stale bot commented Dec 1, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Dec 1, 2022
@yolabingo
Copy link
Contributor Author

I think this will be helpful

@stale stale bot removed the stale label Dec 1, 2022
@yolabingo
Copy link
Contributor Author

support ticket https://dotcms.zendesk.com/agent/tickets/115354

@freddyDOTCMS freddyDOTCMS self-assigned this May 8, 2024
freddyDOTCMS added a commit that referenced this issue May 10, 2024
@freddyDOTCMS
#22033 Creating Util API Level Method to prune all the TimeMachine fo…
b58d063 
…lders olders than PRUNE_TIMEMACHINE_OLDER_THAN_DAYS
freddyDOTCMS added a commit that referenced this issue May 10, 2024
@freddyDOTCMS
#22033 Delete not empty folders
b2069f2
freddyDOTCMS added a commit that referenced this issue May 10, 2024
@freddyDOTCMS
#22033 Using FileUtil.deleteDir method
e7fbab3
@freddyDOTCMS freddyDOTCMS mentioned this issue May 10, 2024
Closed
1 task
freddyDOTCMS added a commit that referenced this issue May 10, 2024
@freddyDOTCMS
#22033 Create util method to remove old Time Machine backups files
df2edfe
@freddyDOTCMS freddyDOTCMS mentioned this issue May 10, 2024
Merged
@freddyDOTCMS freddyDOTCMS added the on hold In Progress, but paused to work on higher priority card label May 13, 2024
github-merge-queue bot pushed a commit that referenced this issue May 13, 2024
@freddyDOTCMS
#22033 Create util method to remove old Time Machine backups files (#…
b1dde74 
…28530)

This is a first PR for #22033

Proposed Changes
Create a Util method to prune old time TimeMachine folder


https://github.com/dotCMS/core/pull/28530/files#diff-5d7a647e09c3a207a6c502bdb945879b865d3dc8ad0cd7cc8c3b4fa233edd3aeR36

Checklist
 Tests
Additional Info
Later, I am going to create a Job to call this method every week
freddyDOTCMS added a commit that referenced this issue May 14, 2024
@freddyDOTCMS
#22033 Creating the PruneTimeMachineBackup Class
aaf446f
@freddyDOTCMS freddyDOTCMS mentioned this issue May 14, 2024
Merged
freddyDOTCMS added a commit that referenced this issue May 15, 2024
@freddyDOTCMS
#22033 Create getAvailableTimeMachineFolder method
d582f88
@freddyDOTCMS freddyDOTCMS mentioned this issue May 15, 2024
Merged
1 task
github-merge-queue bot pushed a commit that referenced this issue May 15, 2024
@freddyDOTCMS
#22033 Creating the PruneTimeMachineBackup Class (#28564)
9f8874c 
This is a second PR for #22033

First one:

#28530

### Proposed Changes
* Create the Job class


https://github.com/dotCMS/core/pull/28564/files#diff-ad7f5fa58328ce7220001bad020c4f3d203dcf182ff280b694f640179483b570R23

### Checklist
Tests

###  Additional Info
Later, I am going to send more changes to start and stop this Job
freddyDOTCMS added a commit that referenced this issue May 16, 2024
@freddyDOTCMS
#22033 Start The PruneTimeMachineJob
ac2b0e8
@freddyDOTCMS freddyDOTCMS mentioned this issue May 16, 2024
Merged
1 task
github-merge-queue bot pushed a commit that referenced this issue May 16, 2024
@freddyDOTCMS
#22033 Create getAvailableTimeMachineFolder method (#28574)
59b38a3 
This is a third PR for #22033

First one: #28530
Second one: #28564

### Proposed Changes

-  Create the getAvailableTimeMachineFolder Method


https://github.com/dotCMS/core/pull/28574/files#diff-5d7a647e09c3a207a6c502bdb945879b865d3dc8ad0cd7cc8c3b4fa233edd3aeR53

- I realized that the TomeMachine Job created the folder with the "tm_"
preffix instead of "timeMachineBundle" preffix, so I have to update all
the test:


https://github.com/dotCMS/core/pull/28574/files#diff-5ec720d6f4deb8a5228aee22286a6bbf3103289becc6d1fa4947017c9052f935L66-R151


https://github.com/dotCMS/core/pull/28574/files#diff-f27467cb9919462f8c7c5a137c3c9b9bf21be349047c0d1e1ba9315773e53298L43-R59

- Turn the "tm_" preffix into a Constant


https://github.com/dotCMS/core/pull/28574/files#diff-19b28e2982b6db869240fb8fc96a2bf2c156e3607f41656fd16d7845c45ba604R37

and use it to create a FileFilter


https://github.com/dotCMS/core/pull/28574/files#diff-19b28e2982b6db869240fb8fc96a2bf2c156e3607f41656fd16d7845c45ba604R245-R251

- The notRemoveNotTimeMachineFolder test was wrong because the
"TIMEMACHINE_PATH" property was not being set on the test:


https://github.com/dotCMS/core/pull/28574/files#diff-f27467cb9919462f8c7c5a137c3c9b9bf21be349047c0d1e1ba9315773e53298R132

### Checklist

- [x] Tests

### Additional Info
Later, I am going to use this method on the
[PruneTimeMachineBackupJob.java](https://github.com/dotCMS/core/pull/28564/files#diff-ad7f5fa58328ce7220001bad020c4f3d203dcf182ff280b694f640179483b570)
to know when it need to be stopped, this job should be started when the
TimeMachineJob is started and should be finished after the
TimeMachineJob is stopped and no more Time Machine Folder are pending to
be cleaned up
dsilvam added a commit that referenced this issue May 22, 2024
@dsilvam
#22033 schedule prune job from the beginning
90770c7
freddyDOTCMS added a commit that referenced this issue May 22, 2024
@freddyDOTCMS
#22033 Including folder in bundles path to be removed
9730bd5
github-merge-queue bot pushed a commit that referenced this issue May 23, 2024
@freddyDOTCMS @dsilvam
#22033 Start The PruneTimeMachineJob (#28586)
554104a 
This is a last PR for #22033

First one: #28530
Second one: #28564
third one: #28586

On this PR we are starting the PruneTimeMachineJob, this need to be
starter when the TimeMachineJob is started.

### Proposed Changes
* Start the Prune Time machineJob after start the TimeMachineJob


https://github.com/dotCMS/core/pull/28586/files#diff-d6eb70242764e7e3dbcffc3d0bac20c485b0879b72118925714bb30e37d1ac9aR507

* We need to remove the files under the bundle folder too


https://github.com/dotCMS/core/pull/28586/files#diff-19b28e2982b6db869240fb8fc96a2bf2c156e3607f41656fd16d7845c45ba604R231

When TimeMachine run it created the Bundle folder under the bundles
folder and later copy it to the timemachine folder, we need to remove
both

### Checklist
- [x] Tests
- [ ] 
### Additional Info
Later I am going to sent a last PR with the Stop logic

---------

Co-authored-by: Daniel Silva <daniel.silva@dotcms.com>
@freddyDOTCMS freddyDOTCMS removed their assignment May 23, 2024
@freddyDOTCMS freddyDOTCMS added Merged and removed on hold In Progress, but paused to work on higher priority card labels May 23, 2024
@freddyDOTCMS
Copy link
Contributor

PR: #28586

@dsilvam dsilvam self-assigned this May 24, 2024
dsilvam added a commit that referenced this issue May 24, 2024
@dsilvam
#22033 log backups removd
48c33e0
github-merge-queue bot pushed a commit that referenced this issue May 24, 2024
@dsilvam
#22033 log backups removd (#28665)
e9e6fe5 
Include logging of backups deleted
This was linked to pull requests May 24, 2024
#22033 Start The PruneTimeMachineJob #28586
Merged
#22033 log backups removd #28665
Merged
@dsilvam
Copy link
Contributor

dsilvam commented May 24, 2024

Passed Internal QA: Job properly deleting backups older than the specified number of days.

@dsilvam dsilvam removed their assignment May 24, 2024
@josemejias11
Copy link
Contributor

Approved: Tested on trunk_f28fc49, Docker, macOS 13.0, FF v124.0.2

spbolton pushed a commit that referenced this issue Jun 3, 2024
@freddyDOTCMS @dsilvam @spbolton
#22033 Start The PruneTimeMachineJob (#28586)
747e418 
This is a last PR for #22033

First one: #28530
Second one: #28564
third one: #28586

On this PR we are starting the PruneTimeMachineJob, this need to be
starter when the TimeMachineJob is started.

### Proposed Changes
* Start the Prune Time machineJob after start the TimeMachineJob


https://github.com/dotCMS/core/pull/28586/files#diff-d6eb70242764e7e3dbcffc3d0bac20c485b0879b72118925714bb30e37d1ac9aR507

* We need to remove the files under the bundle folder too


https://github.com/dotCMS/core/pull/28586/files#diff-19b28e2982b6db869240fb8fc96a2bf2c156e3607f41656fd16d7845c45ba604R231

When TimeMachine run it created the Bundle folder under the bundles
folder and later copy it to the timemachine folder, we need to remove
both

### Checklist
- [x] Tests
- [ ] 
### Additional Info
Later I am going to sent a last PR with the Stop logic

---------

Co-authored-by: Daniel Silva <daniel.silva@dotcms.com>
spbolton pushed a commit that referenced this issue Jun 3, 2024
@dsilvam @spbolton
#22033 log backups removd (#28665)
b42a58c 
Include logging of backups deleted
@dsilvam dsilvam self-assigned this Jun 5, 2024
@dsilvam dsilvam closed this as completed Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dsilvam dsilvam

Labels
Merged OKR : Customer Support Owned by Scott OKR : Technical User Experience Owned by Freddy Priority : 3 Average QA : Approved QA : Passed Internal Release : 24.05.31 Bug Fixing Team : Falcon Type : Enhancement
Projects
dotCMS - Product Planning
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#22033 Start The PruneTimeMachineJob dotCMS/core
#22033 log backups removd dotCMS/core
8 participants
@yolabingo @wezell @dsilvam @erickgonzalez @swicken-dotcms @damen-dotcms @josemejias11 @freddyDOTCMS