← Back to Dashboard
1. When to Customize a Model2. Evaluation of Customized Models
Evaluation of Customized Models
📚 Fine-Tuning and Customization⏱ 11 min⭐ 90 XP
Quality and Safety Gatekeeping
Evaluate customized models on task quality, safety behavior, latency, and cost impact. A quality gain that increases risk is not a production win - fine-tuning can shift safety behavior in ways base-model evaluations never covered.
The Four-Way Comparison
A customized model must beat all realistic alternatives, not just its own base model:
| Alternative | Question it answers |
|---|---|
| Base model + your best prompt | Was prompting alone enough? |
| Base model + RAG | Was missing knowledge the real gap? |
| A stronger off-the-shelf tier | Would upgrading the model beat customizing this one? |
| Your fine-tune | Does customization win after Provisioned Throughput cost? |
Evaluation Protocol
- Held-out test sets - never evaluate on training data; keep a frozen split the training pipeline cannot see.
- Adversarial prompts - fine-tuning can erode refusal behavior; rerun the full safety suite, not just task metrics.
- Regression breadth - test adjacent tasks too: a model tuned for triage summaries may get worse at general Q&A your app also uses.
- Latency + cost - measure full with Provisioned Throughput economics included, not just per-token price.
Post-Deployment Drift
Track drift after deployment: input distributions shift, and a model tuned on last quarter's ticket styles slowly degrades. Schedule periodic re-evaluation on fresh samples and define a retirement/retraining trigger (e.g. task success −5% versus launch baseline).
Decision rule: promote a custom model only when it wins the four-way comparison on quality and shows zero safety regressions and the total cost of ownership (training + Provisioned Throughput + re-evaluation cadence) beats the best alternative. Two out of three is a rejection.
🧪 Knowledge Check
Press 1-4 to select1 of 2
A valid model customization launch requires:
Higher quality only
Quality + safety + latency + cost validation
No baseline comparison
No monitoring