Our QA fine-tune, aligned with classic RLAIF: a Bradley-Terry reward model trained on 500 AI preferences, then PPO against it with a KL anchor to the reference.
The QA-SFT checkpoint, aligned by reinforcement learning against a reward model. The reward model — a scalar head on the 500M backbone — reached 0.983 held-out pairwise accuracy on the preferences. PPO then optimized the policy toward higher reward while a KL penalty kept it anchored to the reference (reward rose, KL stayed bounded).
60 PPO iterations, ~21 min on an L4. Judged correctness 0.443 [0.406–0.480] — the strongest of our from-scratch models and a significant gain over 500M DPO (0.279). The reward model reached 0.983 held-out pairwise accuracy.
| Class | LlamaForCausalLM |
| Layers | 24 |
| Hidden size | 1,280 |
| Attention | 20 heads · dim 64 · full MHA |
| Feed-forward | SwiGLU · inner 3,456 |
| Positional | RoPE · θ 10,000 |
| Norm | RMSNorm · ε 1e-5 |
| Context | 1,024 tokens |
| Vocabulary | 32,768 |
| Embeddings | tied input/output |
| Init from | slm-500m QA-SFT |
| Method | RLAIF — reward model + PPO |
| Trainable params | 517.8M — all |
| Training data | 500 preference pairs |
| Training tokens | 325K per PPO pass |
| PPO iterations | 60 |
our cost begins at fine-tuning — the base is imported.
| Stage | Detail | Cost |
|---|---|---|
| QA supervised fine-tune | the checkpoint RLAIF starts from | $0.93 |
| Reward model | Bradley-Terry head on L4 | $0.02 |
| PPO alignment | 60 iterations on L4 | $0.33 |
| Evaluation (shared) | 13 versions on 500 held-out questions, this model's share | $1.48 |
| Total | $2.76 |
Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.