Dummy token for prompts in HH datasets

[louieworth]

I am issue regarding the Anthropic/hh-rlhf dataset, in reward_dataset.py:

# Anthropic/hh-rlhf
# tasksource/oasst1_pairwise_rlhf_reward
if exist_and_not_none(data, "chosen") and exist_and_not_none(data, "rejected"):
    prompt = data["prompt"] if exist_and_not_none(data, "prompt") else ""
    if prompt.startswith("prompter:"):
        prompt = (
            prompt.replace("prompter:", "\nHuman:\n").replace("assistant:", "\nAssistant:\n")
            + "\nAssistant:\n"
        )
    chosen = data["chosen"]
    reject = data["rejected"]
    input_template = None  # do not modified with input template again

This will result in prompt="" for every sample as the Anthropic/hh-rlhf = {'chosen', rejected}`. Will this cause an issue in accounting the logits of prompts in the following code:

OpenRLHF/openrlhf/trainer/dpo_trainer.py

Line 301 in fbc1eab

mask[:source_len] = False

[hijkzzz]

Theoretically, whether mask prompt or not has no effect on DPO loss.

[louieworth]

It seems true as the prompt logits will be canceled by (chosen_logits - rejected_logits). Will this affect other algorithms? such as KTO?