Reinforcement Fine-Tuning & DPO — Fine-Tuning & Distillation

Beyond Supervised Fine-Tuning

Reinforcement Fine-Tuning (RFT) goes beyond traditional supervised fine-tuning by using verifiable rewards to train models on tasks where correctness can be programmatically checked - such as code execution, math proofs, or structured data extraction.

How RFT Works

1. Define a Grader: Write a programmable grader that scores model outputs (pass/fail, 0-1 score, or multi-criteria rubrics).
2. Submit Training Prompts: Provide a set of prompts without reference answers - the model learns by generating candidates and receiving reward signals.
3. Reinforcement Loop: The model iteratively improves, maximizing the grader's reward function. Unlike SFT, the model discovers its own optimal strategies.

Programmable Graders

Graders are the core innovation. They can be simple (exact string match), moderate (unit test execution), or complex (LLM-as-judge with rubric). OpenAI provides built-in grader templates:

// RFT training configuration (supported on reasoning models like o4-mini)
const job = await openai.fineTuning.jobs.create({
  training_file: "file-prompts123",
  validation_file: "file-val123",
  model: "o4-mini-2025-04-16",
  method: {
    type: "reinforcement",
    reinforcement: {
      grader: {
        type: "python",  // Programmatic Python grader (or string_check, score_model, multi)
        source: "def grade(sample, item):
  return 1.0 if sample['output_text'].strip() == item['reference_answer'].strip() else 0.0"
      }
    }
  }
});

Direct Preference Optimization (DPO)

DPO is a complementary technique for preference alignment - teaching the model which of two outputs is preferred by humans. Instead of a scalar reward, you provide pairs of (chosen, rejected) responses. DPO adjusts the model to increase the likelihood of chosen outputs and decrease rejected ones, without needing a separate reward model.

When to Use Each