Data inconsistency between DB & ES

[cobolbaby]

Describe the bug
When I delete some dataset description, the description information in the database(MySQL/PG) could be cleaned, but is not deleted synchronously in ES .

To Reproduce
Steps to reproduce the behavior:

1. Set the dataset description
2. Check data inconsistency between DB & ES
3. Clean the dataset description as same above
4. Check data inconsistency between DB & ES

Expected behavior
Data needs to be synchronized between different databases, otherwise the search results may be incorrect.

[mars-lan]

Could you elaborate how you "Set the dataset description" in step 1?

[cobolbaby]

I just import dataset using etl script in metadata-ingestion directory.

[mars-lan]

I see. In that case MAE should be fired and processed by MAE processor job which updates the search index

[mars-lan]

Actually after reading your steps more closely again, I think I understand what you meant now. I assume you mean that @ Step 2 it is consistent but @ Step 4 it is not?

[cobolbaby]

Yes. When I just modify the content of the description, the data can be kept in sync. But if I perform a delete operation, there is a problem with data consistency.

[mars-lan]

I assume you manually delete the entry from DB? That won't go through the DAO and thus no MAE emitted.

[cobolbaby]

I just deleted the description of the schema.

After I imported the dataset again, I saw the new version of the dataset in the pg.

[mars-lan]

Looks like the builder will only partial update description when there's one: https://github.com/linkedin/datahub/blob/master/metadata-builders/src/main/java/com/linkedin/metadata/builders/search/DatasetIndexBuilder.java#L73

Need to explicitly set it to null when there's no description.

[cobolbaby]

Do you mean to pass a "null" string ? Or just keep using the current code https://github.com/linkedin/datahub/blob/master/metadata-ingestion/sql-etl/common.py#L56

In my opinion, storing a null string may not be friendly.

[mars-lan]

If you change that from None to empty string ("") it would most likely fix your problem. The proper fix IMO is still in DatasetIndexBuilder. I'll create a PR tomorrow to fix this.