Potential performance issue: .to_dict("records") method slow in pandas below 2.0

[TendouArisu]

Issue Description:

Hello.
I have discovered a performance degradation in the .to_dict("records") function of pandas version below 2.0. And I noticed that some parts of the repository depend on the pandas version 1.3.5 such as scripts/duckdbless_requirements.txt. I found that many files such as splink/term_frequencies.py and splink/expectation_maximisation.py used the influenced api. There may be more files using the influenced api. I am not sure whether this performance problem in pandas will affect this repository. Here are some discussions on pandas GitHub related to this issue, including #46470 and #46487.

Suggestion

I would recommend considering an upgrade to a different version of pandas >= 2.0 or exploring other solutions to optimize the performance.
Any other workarounds or solutions would be greatly appreciated.
Thank you!

[RobinL]

Thanks for the report. Splink 4 will use pandas 2.0, and our lockfile in splink3 now also uses >2.0 so i think this can be closed now