GH-5732: Make TypeEngine.lazy_import_transformers() thread safe

[Tom-Newton]

Tracking issue

Closes: flyteorg/flyte#5732

Why are the changes needed?

Lack of thread safety in TypeEngine.lazy_import_transformers() can cause errors when triggering Flyte executions simultaneously from concurrent threads.

What changes were proposed in this pull request?

• Add a lock to make it threadsafe.
  • I considered just moving cls.has_lazy_import = True to the end of the function but I decided against it for 2 reasons:
    • Its possible that importing transformers twice could cause issues - though currently I don't think it does.
    • Apparently python variable assignments are not always atomic https://github.com/google/styleguide/blob/91d6e367e384b0d8aaaf7ce95029514fcdf38651/pyguide.md#218-threading - though in practice I doubt this will be a problem.
• Write a test

How was this patch tested?

Wrote a test

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly. - not applicable
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.55%. Comparing base (9f9f197) to head (8a4b484).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2735   +/-   ##
=======================================
  Coverage   83.55%   83.55%           
=======================================
  Files           3        3           
  Lines         152      152           
=======================================
  Hits          127      127           
  Misses         25       25           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pingsutw]

Thank you @Tom-Newton

[eapolinario]

LGTM.