Reorganize ingest_metadata field

[shirey]

The Dataset.ingest_metadata field will be reorganized to accommodate:

• Easier access to metadata for searching
• Adding fields to accommodate the Workflow Tracking effort
• Adding the Dataset.calculated_metadata field to allow storage of calculated/derived metadata

This work is specified in the ingest_metadata field reorganization document

From the above document a summary of the changes that need to be made are:

• MOVE:

  • Dataset.ingest_metadata.metadata → Dataset.metadata
  • Dataset.ingest_metadata.files → Dataset.files

• DELETE:

  • Dataset.ingest_metadata.thumbnail_file_abs_path
  • Dataset.ingest_metadata.extra_metadata

• ADD:

  • Dataset.ingest_metadata.workflow_description (as String/Text)
  • Dataset.ingest_metadata.workflow_version (as String/Text)
  • Dataset.calculated_metadata (as JSON text)

• RENAME (Elasticsearch/Search API only):

  • Dataset.metadata → Dataset.ingest_metadata (Elasticsearch/Search API only)

[tjmadonna]

Relevant SenNet PR

sennetconsortium/entity-api#510

Script for updating datasets

https://gist.github.com/tjmadonna/48f7d00d84ee0b36094bf337e7cabc87