Implement SeaDatabricksClient (Complete Execution Spec)

[varun-edachali-dbx]

What type of PR is this?

• Feature

Description

• Implement execution relevant methods in SeaDatabricksClient as defined in the DatabricksClient interface.
• Does not implement metadata relevant methods.

How is this tested?

• Unit tests
• E2E Tests
• Manually - using the test scripts to be introduced in Introduce manual SEA test scripts for Exec Phase  #589
• N/A

The coverage of SeaDatabricksClient by test_sea_backend.py's unit tests are as below.

Module	Statements	Missing	Coverage	Notes
backend.py (SeaDatabricksClient class)	165	2	99%	Missing lines are TYPE_CHECKING imports (lines 11-12) which are not meant to be covered in runtime tests

Related Tickets & Documents

https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?usp=sharing

[samikshya-db]

why is this present in sea_backend test?

[varun-edachali-dbx]

It's used to test how some methods respond to invalid session id's (i.e., ones constructed from thrift handles instead of sea).

[samikshya-db]

Can you explore if @dataclass makes this JSON deserialising cleaner?

[varun-edachali-dbx]

Based on some exploration, there seem to be ways to serialise and deserialise dataclasses into JSON less explicitly, but I don't think it's a good idea for us:

1. If we're looking for direct conversion (i.e., just ClassName(**json)): we cannot directly unpack nested JSON into nested dataclasses, which is a common use case for us (Chunks in ManifestData) - [StackOverflow discussion].
2. Sometimes we want to convert the values into specific forms (with non-trivial conversion methods) before we store them in our types, eg: the state needs to be stored as CommandState in our models. We would have to define custom methods for this serialisation and de-serialisation, which can get quite verbose.

There are ways to help mitigate both of the above complexities, mainly by using third-party libraries, but I think the code in it's current form (with explicit conversion) is more readable.

[github-actions]

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).