Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Low (would be nice)

Please provide a clear description of problem this feature solves

This issues arises from the discussion in this thread.

Some settings in Morpheus are only configurable via an environment variable (DATABRICKS_HOST). Other settings have pipeline-specific settings but can be overriden by environment variables (MORPHEUS_FILE_DOWNLOAD_TYPE). This method of overriding forces all stages to use the same values, meaning we cannot write to two separate databricks hosts, and if the MORPHEUS_FILE_DOWNLOAD_TYPE is set, we cannot download different types of files per stage.

Is environment-variable-based overrides an intentional feature of Morpheus, or simply a byproduct of implementation? If the latter, we should consider providing an alternate solution to the problem of overriding pipeline-specific settings. If the former, we should ensure all new functionality follows this paradigm.

Describe your ideal solution

A per-stage or per-instance way of overriding stage-specific settings from the command line for a pre-existing morpheus pipeline (one declared in a run.py, for instance).

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request