Explore Alternative ML Model Providers

[davidgxue]

Please describe the feature you'd like to see
During downtimes of services such as OpenAI, Cohere and Azure OpenAI, have hot swappable.

For cost reduction, we should also do A/B testing and see how other models perform compared to GPT-4 as GPT-4 itself is very expensive.

Describe the solution you'd like
Explore using OpenAI and Anthropic for LLM services as backup for Azure OpenAI's GPT-3.5 and GPT-4.

Are there any alternatives to this feature?
Explore open source LLMs

Additional context

Acceptance Criteria

• Implement env variable based mechanism to hot swap an existing LLM to another provider during service downtime