[Feature]: Dynamic tpm quota (multiple projects)

[krrishdholakia]

The Feature

Allow setting a dynamic_quote: true flag for a dynamic tpm budget dependent on how many simultaneous apps there are

Motivation, pitch

prevent apps from exhausting tpm quota

Twitter / LinkedIn details

cc: @jeromeroussin

[krrishdholakia]

for example you'd have to define how long an app is considered "active" after their last call, so you can accurately compute a number of "simultaneous" apps

[krrishdholakia]

can we do this similar to latency ttl?

check the number of active apps -> divide the available tpm capacity by number of active apps -> return that as the available tpm for that call

[krrishdholakia]

this probably wouldn't be in router.py

as it doesn't really have anything to do with the load balancing between models

[krrishdholakia]

would probably be a call hook on the proxy

similar to parallel_request_limiter.py -

litellm/litellm/proxy/hooks/parallel_request_limiter.py

Line 4 in c0540e7

from litellm.proxy._types import UserAPIKeyAuth

[krrishdholakia]

Basic test cases:

• If 1 'active' project => give all tpm
• If 2 'active' projects => divide tpm in 2

[krrishdholakia]

For v0 considering tracking 'active apps' by model with ttl = 60s

[krrishdholakia]

E2E case - "I am more concerned about apps gobbling up all the quota"

base case:
1 project -> gets all quota

advanced case:
2 projects -> each gets equal quota