Skip to content

Remove usage of pickle from objectdb and history #605

@lieryan

Description

@lieryan

pickle is insecure and it's not very efficient for larger, more complex datasets.

This will be a multiple steps plan towards moving into sqlite-backed storage:

  1. Remove rarely used features
    1. Deprecate objectdb and history compression: This was never the default and disk space nowadays makes this unlikely to be an issue nowadays, maybe we can re-introduce compression when sqlite migration is complete (Remove .ropeproject/history and .ropeproject/objectdb compression #606)
    2. Deprecate importing of old format (objectdb.pickle and history.pickle): rope hasn't been generating this for decades (Remove _DataFiles._import_old_files() #607)
  2. Make ScopeInfo JSON-serializable: there are some data here that are python class instances, but they really doesn't need to be, we can just __getstate__()/__setstate__() them instead when de/serializing. Implement JSON DataFile serialization  #611
  3. Implement sqlite database schema versioning Automatically purge autoimport.db when there is schema change  #565
  4. These files contains a bunch of nested dictionaries, these can be translated to sqlite tables in a straightforward manner, we'll likely will still store JSON documents inside the sqlite tables instead of normalizing everything
  5. Make sure that the tables have appropriate indexes
  6. Remove write support for legacy objectdb and history files
  7. Remove read support for legacy objectdb and history files

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions