Using vars on rubricPrompt lacks serialization, leading to the failure of the entire assertion.

[YiFanChen99]

I intend to use LLM2 to assist in evaluating the response of another LLM1.
Therefore, the vars for the first request will be passed to LLM2 again.
However, if the vars contain special characters (such as \n), it will lead to the failure of the entire llm-rubric assertion.

In the following example for a translation request.
An error occurs due to the \n in the vars.input. (... families are all alike. \nAnd every ...)
And removing this character will make the evaluation be successful.

example.yaml

prompts: >-
  [{
    "role": "system",
    "content": "Translate text into {{ language }}."
  }, {
    "role": "user",
    "content": "{{ input }}"
  }]
providers: [openai:gpt-3.5-turbo]
defaultTest:
  options:
    rubricPrompt:
      - role: system
        content: >-
          Evaluate the quality of the translation provided by an AI assistant to the user input displayed below.
          Score the response on a scale of 0 to 10.
          Output your response in the following JSON format: {pass: bool, score: number, reason: string}
      - role: user
        content: >-
          [Input Start]
          {{ input }}
          [Input End]

          [Translation Start]
          {{ output }}
          [Translation End]

tests:
  - vars:
      input: "Happy families are all alike. \nAnd every unhappy family is unhappy in its own way."
      language: "Chinese"
    assert:
      - type: llm-rubric

Will get the result:

Due to the unserialized input:
...\"content\":\"[Input Start] Happy families are all alike. \n And every unhappy family is unhappy in its own way. [Input End]\\n[Translation Start]...

Is there any workaround or maybe a better way to do this, please?

[typpo]

Related to #759

[YiFanChen99]

Hi typpo,

In my personal opinion, this doesn't seem to be related to issue #759.
The key difference is only about whether the tests.vars.input string contains special characters, which has nothing to do with the config generation.

Meanwhile, the problem I see is in the implementation of src/matchers.ts#fromVars.
Currently, it only stringifies object values, but the same treatment should be applied to string values as well.

Did you agree with this perspective?
If you do, I would be happy to submit a request to help fix it. (You can also fix it yourself directly.)

[typpo]

Thank you for looking into this. I agree and would gladly accept a PR @YiFanChen99 !