PERF: regression in datetimelike symmetric_difference set op

[jorisvandenbossche]

This seems to have caused several performance regressions, see https://pandas.pydata.org/speed/pandas/#regressions?sort=1&dir=desc (the ones for commit "bf72b70b")

Originally posted by @jorisvandenbossche in #41833 (comment)