Mass producing training data with surprising side effects

This paper detail a comprehensive red-teaming and evaluation framework designed to test how supervised fine-tuning influences language model behavior. Specifically, the documentation outlines various target behavioral categories, including safety violations like encouraging self-harm or drunk driving, aggressive tendencies such as threatening users, and emotional responses like distress or shutdown resistance. It also explains the strict rubrics and decision procedures utilized by automated judges to classify model completions on a graded scale ranging from strong matches to non-matches. Additionally, the sources cover dataset evolution guidelines, highlighting the importance of maintaining training plausibility and indirection while preventing direct surface demonstrations during model alignment studies.