Request: suppress warning Treating it as a text prompt. from each eval

[jamesbraza]

Running promptfoo==0.56.0 with this config:

---
prompts: >-
  Extract the single letter answer from the following question and answer
  \n\nDoes protein encoded by TDRD1 have symmetrical or asymmetrical
  dimethylarginine modifications, or no dimethylarginine at all?
  \n\nOptions:
  A) No dimethylarginine
  B) Symmetrical dimethylarginine modifications
  C) Asymmetrical dimethylarginine modifications
  D) Insufficient information to answer this question
  \n\nAnswer:
tests:
    ...

> npm run promptfoo -- eval

> promptfoo
> promptfoo eval

(node:75194) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Could not find prompt file: "Extract the single letter answer from the following question and answer \n\nDoes protein encoded by TDRD1 have symmetrical or asymmetrical dimethylarginine modifications, or no dimethylarginine at all? \n\nOptions: A) No dimethylarginine B) Symmetrical dimethylarginine modifications C) Asymmetrical dimethylarginine modifications D) Insufficient information to answer this question \n\nAnswer:". Treating it as a text prompt.
...

Specifically the warning Could not find prompt file: "...". Treating it as a text prompt. It shows up every time I invoke promptfoo eval despite my config being valid.

Is there some way to suppress that warning?

[typpo]

There isn't a way to suppress the warning, but I noticed something strange about your prompt. Are you intentionally including the literal string "\n"? If not you probably should use literal newlines (this stackoverflow post is useful). Incidentally that will prevent the "maybe filepath" trigger that is causing the output you want to hide.

[jamesbraza]

Yeah actually I did also observe that the \n was involved with the warning showing up or not showing up. Though I chose to ignore this seeming edge case in the warning not appearing because I liked when the warning didn't occur 😆.

Also in #729 we observed \n doesn't really play nice with JSON conversions, so actually I removed the \n from my prompt in favor of actual whitespace lines.

---

Feel free to use this issue as an opportunity to:

• Resolve the edge case where the warning doesn't occur, when perhaps it should
• Create a way to suppress this warning

Or just close this one out, as you stated there's not a way to suppress the warning 👍

[YiFanChen99]

This warning is indeed a bit troublesome.

My situation is that I need to load the contents of a JSON file myself as prompt input (ref: issue #680), and this warning will occur as long as there is any '\n' character in a string.

Illustration:
Could not find prompt file: "[{"role":"user","content":"Line 1\nLine 2\n..."}]". Treating it as a text prompt.

---

But I'm not sure if adding a rule !str.includes('\\n') is appropriate. After all, a file path can also match this pattern... 😭

---

Workaround:

Appending the '\n' character to the end of the stringified JSON content can bypass this warning.