Delete and create the same changefeed in a short time may cause deadlock and data loss

[overvenus]

What did you do?

Because of model.ChangeFeedID is non-unique, owner and process has no way to
find out whether they are running the same changefeed. After appling delete and
create operation on the same changefeed name, there is a race condition between
owner and processor that they are running two different changefeeds. This may
cause deadlock and data loss.

To fix the issue, we should make sure:

1. When owner receives a delete changefeed request, it can only return a
   response after the changefeed is closed completely on every TiCDC node. It
   prevents data loss.
2. Add a unique field to model.ChangeFeed. It prevents deadlock and other
   illegal states.

What did you expect to see?

No deadlock and no data loss.

What did you see instead?

Changefeed is a deadlock state, and lag is increasing.

Versions of the cluster

TiCDC version (execute cdc version):

All TiCDC released version

[AkiraXie]

/found automation

[AkiraXie]

/severity major

[nongfushanquan]

/assign @overvenus

[nongfushanquan]

#7730

[overvenus]

This issue has not been fixed, #7730 is a workaround that only mitigates the issue.

[overvenus]

Find a similar case, restart changefeed in a very short period. Owner may see processor's state pre-restart changefeed.

See also #8242