CV3-47 Use celery to delete archived data that is over 30 days old #510
Conversation
mifeille
force-pushed
the
story/CV3-47-delete-archived-data
branch
9 times, most recently
from
113a74d to
89c5f49
Compare
mifeille
force-pushed
the
story/CV3-47-delete-archived-data
branch
from
89c5f49 to
cd4b237
Compare
mifeille
force-pushed
the
story/CV3-47-delete-archived-data
branch
2 times, most recently
from
5441f22 to
fe09586
Compare
|
|
||
| @celery.task(name='clean_archived_data.delete_archived_data') | ||
| def delete_archived_data(): | ||
| database_engine = create_engine(database_uri) |
There was a problem hiding this comment.
would you please put in a docstring which explains what this function, specifically to delete data which are 30 days old with also an archived state.
| now = datetime.now() | ||
| delta = now - timedelta(days=30) | ||
| statement = table.delete().where( | ||
| and_(table.c.date_updated < delta, table.c.state == 'archived')) |
There was a problem hiding this comment.
@mifeille after finishing the job, this is what celery.err.log is saying sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) update or delete on table "rooms" violates foreign key constraint "room_tags_room_id_fkey" on table "room_tags" DETAIL: Key (id)=(1) is still referenced from table "room_tags". [SQL: 'DELETE FROM rooms WHERE rooms.date_updated < %(date_updated_1)s AND rooms.state = %(state_1)s'] [parameters: {'date_updated_1': datetime.datetime(2019, 10, 5, 14, 31, 1, 378610), 'state_1': 'archived'}] (Background on this error at: http://sqlalche.me/e/gkpj)
There was a problem hiding this comment.
There is a migration to include cascading deletion I guess that one on room tags was forgotten, I am going to add it.
mifeille
force-pushed
the
story/CV3-47-delete-archived-data
branch
11 times, most recently
from
25d2812 to
0e2b3fd
Compare
mifeille
force-pushed
the
story/CV3-47-delete-archived-data
branch
3 times, most recently
from
0b05269 to
7216ae3
Compare
joshuaocero
left a comment
joshuaocero
left a comment
There was a problem hiding this comment.
@mifeille this looks really good. Please remove the print statement so I can merge.
| database_engine = create_engine(database_uri) | ||
| metadata = MetaData() | ||
| metadata.reflect(bind=database_engine) | ||
| print('deleting archived data...') |
There was a problem hiding this comment.
Please remove this debug statement.
- configure celery beat - implement use celery to delete archived data that is over 30 days old [Delivers CV3-47]
mifeille
force-pushed
the
story/CV3-47-delete-archived-data
branch
from
7216ae3 to
8df72b2
Compare
Description
Currently, the Backend implements soft deletes which instead of wiping a record from the database, changes the state of a row from active to archived. We would like to periodically remove rows whose state is archived from the database.
Type of change
Please select the relevant option
Checklist
JIRA
CV3-47