[Feature suggestion]: Exposing model costs for usage with Custom Providers

[Codeshark-NET]

Currently I have code like this, hardcoding the model cost into my custom provider, and returning the total cost that way.

// ...

const inputCost = 0.00025 / 1000; //@TODO: Get from provider instead?
const outputCost = 0.00125 / 1000; //@TODO: Get from provider instead?
const totalCost = inputCost * usage.promptTokens + outputCost * usage.completionTokens || undefined;

return {
  // Required
  output: object,

  // Optional
  tokenUsage: {
    total: usage.totalTokens,
    prompt: usage.promptTokens,
    completion: usage.completionTokens
  },

  cost: totalCost
};

From what I can see, it's not exposed currently, but it'd be super cool if I would be able to look up the model name via the provider, "claude-3-haiku-20240307", and get the cost model back to be used in the custom provider like this. 🙌

[SysOverdrive]

Hey Codeshark,

Cool feature request !

I was also wondering how you accomodated the custom provider did you use the python guide or the Custom Scripts

[Codeshark-NET]

Hey Codeshark,

Cool feature request !

I was also wondering how you accomodated the custom provider did you use the python guide or the Custom Scripts

Heya @SysOverdrive 🙏 you can check out the full code I'm using here: https://github.com/promptfoo/promptfoo/blob/main/examples/custom-provider/vercelAiSdkExample.js (it should be .mjs)

[dynamicwebpaige]

@Codeshark-NET Love this feature request!

Would recommend attempting to extract $/tokens this from the API providers, as opposed to adding manually - the cost for input and output tokens seems to be changing fairly rapidly.