Loadbalancer for model providers as fallback functionality

[AndyMik90]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing?

The new load balancer for model providers gives the ability to use several keys for rate limiting.

2. Describe the feature you'd like to see

Use the same functionality but be able to add different model providers so that if one model provider is down (for instance, when OpenAI was down for many hours, Dify could use a fallback mechanism to use the new API key (with another provider).

3. How will this feature improve your workflow or experience?

Secure that users don't experience downtime if a modelproviders API is down.

4. Additional context or comments

No response

5. Can you help us with this feature?

• I am interested in contributing to this feature.

[AndyMik90]

To clarify, instead of adding 2 Open API keys, we could add 1 Open AI and one Cohere or other model prodiver, to have a fallback if the Open AI API goes down (like it does every now and then).