Multi-version support in master

[akhurana001]

Hi @liyinan926, I am creating this issue to track any open items blocking merging the multi-version-support branch back into master.

The unit-tests were fixed as part of #526 and let's use this issue to track if there is any other work required for this. Happy to pick-up if there are any open items.

[liyinan926]

@pmkc we plan to bump the version to v1beta2 soon with #578. The multi-version-support branch contains backward-incompatible changes so likely will go into v1beta3.

[akhurana001]

Thanks @liyinan926 . #578 is merged. Let me know if there is anything else which needs to be done before we can merge the multi-version as well.

[liyinan926]

I would like to see more users testing this before making it the default in master. We are also doing some testing on our end. I suggest baking it a bit more before merging it back to master. I think we can target mid of Q4 to get it in and released in v1beta3. Does that make sense?

[liyinan926]

@akhurana001 I rebased the branch multi-version-support onto the latest master. If possible, please test using the latest version, which uses the new v1beta2 version of the API.

[akhurana001]

@liyinan926 Thanks for merging. I can definitely run some tests on the latest version. I do want to minimize the number of version migrations for our prod as it becomes a little tricky to migrate to a new version with live workloads due to potential downtime etc. I was also looking at https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/ to see if it makes sense for us to support version conversions.

Overall, We have been running the multi-version-support branch for 6+ months now in production and it has been fairly stable for us.

[urihoenig]

@akhurana001 / @liyinan926 - Is there any corresponding helm chart issue/PR to add job permissions to the clusterrole? (I couldn't find one and thought I might be missing something)

[akhurana001]

@urihoenig That's correct, The helm chart currently is based on the master branch/image of the operator.
@liyinan926 Let's sync-up (maybe a meeting ?) to see how we can make progress on merging multi-version into master. I can also create a PR to update the helm chart as part of that.

[akhurana001]

@liyinan926 circling back on this to see if we can get this done . Happy to chat/help if there are any open issues/tasks blocking this.

[liyinan926]

@akhurana001 thanks for checking on this. We will start creating PRs against the master soon after we get the next release out, which is blocked by the release of Spark 2.4.5.

[akhurana001]

Thanks @liyinan926 . Let me know if there is anything i can help with.

[urihoenig]

Hi @liyinan926 - when is this expected? Is the current multi-version-support rebased on 1.1.0? (which spark version does it require)

[kumare3]

@liyinan926 how should we make it such that we can merge this into master? The problem is we are running this in production and we want to continue adding features. Please help us understand what is required.

[liyinan926]

Nothing major required for this AFAIK. Will start introducing a new API version v1 for this.

[urihoenig]

@liyinan926 - Is this branch still planned to be merged into master? Will it support spark 2 & 3 in parallel?