Direct Optimization Methods

Learn directly from the data without reward modeling (RM) or policy iteration

These models are used to learn the optimal policy directly from the data and are ideal for AI alignment: making AI systems behave in ways that match human values and expectations.

• Reinforcement Learning (RL) methods are often more complex to implement
• RL can be unstable during training
• Direct methods like Direct Preference Optimization (DPO) and Knowledge Transfer Optimization (KTO) can achieve similar or better results with simpler approaches

We want to give feedback that helps them improve and understand what we consider "good" or "bad" responses.