Filebeat 6.3.1 duplicates entries in its registry and reships logs

[frioux]

For confirmed bugs, please report:

• Version: Filebeat 6.3.1
• Operating System: Ubuntu 14.04
• Discuss Forum URL: https://discuss.elastic.co/t/filebeat-6-3-1-update-caused-multiple-entries-in-the-registry-for-the-same-file-logs-re-shipped/140210
• Steps to Reproduce: ???

More details on forum (will copy them here if needed.)

[tsg]

Thank your for the report and the sample registry file, that is really helpful. A fix is currently in progress here: #7632

A workaround, for now is to manually edit the registry file, and:

• Replace meta: null with meta: {} everywhere
• If there is no meta key, add meta: {}

[frioux]

So we actually did that already and restarting filebeat again reintroduced the problem in some places. My coworker is working on a reproduction.

[urso]

Telling from manual testing for the fix, it seems to be related to filebeat upgrading only. On load old registry file entries will get meta set to null. The prospector sets meta to {} by default. For this reason old and new entries do not match. A fresh filebeat installation with empty registry should not be affected by this. For more details and improvements see #7632.