Descope and simplify

[mrmasterplan]

Through years of creep we now have a huge library with a very complicated deployment. This Issue proposes to do the following:

• Support the latest version DBR 14.3 ONLY. All incompatible code will me marked as 'deprecated' and will be untested.
• Change integration tests to use UC and volumes instead of mounts and dbfs. We will not actively remove compatibility with legacy deployments, but it will be untested.
• Change the versioning policy to always follow the DBR version, so it will be 14.3.** for now.
• Always update to latest LTS versions whenever they come out. Any features that are broken by this are not the responsibility of the the person updating the DBR version. These features are marked as deprecated and tests commented out until someone fixes them.

Motivation:

• Integration deployment will be re-written in Terraform for speed and latest features
• Full support of UC and new features is more important than full legacy compatibility.
• Working integration pipeline is more important than updating legacy tests.
• Any end-user who has a problem with any change can lock their spetlr version.

Please respond!

If you are actively monitoring this repo, please leave a comment. Lack of comments will be interpreted as approval.

[FarBo]

I agree with all points. Worths to mention that UC is now enabled by default when a new databricks workspace is created. So it is better to adopting it ASAP, as Databricks is very pushy to migrate to UC for their customers/ users.

We should also include a detailed release note every time that we have minor/ major releases to explain what is changed. We can separate it from readme and add details about changes in the library and deployment, as well as all the bundled features like cosmos, eventhub, sql connections/ drivers, so people know what versions are used when we update them. Then we can ship this release note with the released version.

[LauJohansson]

Fully agree on this!