RLHF reward model rewards verbose answers regardless of correctness

A reward model trained on ~40k preference pairs consistently rates longer responses higher, even when content is wrong. Correlation between reward score and response length is 0.71 on a held-out set. Suspect the annotators (expert contractors) preferred verbose answers.

context

PPO fine-tuning an instruction-tuned Llama model. Reward model shared across three tasks (QA, summarization, code explanation). Preference pairs were collected over 6 weeks.

goal

Propose a concrete set of debiasing steps (data relabeling, length-controlled training, reward-model regularization) that reduces length-correlation to <0.3 while preserving quality judgment ability.

constraints

Cannot re-collect all preference data (budget exhausted). Can collect a small (~2k) corrective set.