Fine-tuned Llama 3.1 70B forgets instruction-following after 800 training steps

Fine-tuning Llama 3.1 70B with QLoRA on ~50k domain-specific examples shows training loss decreasing nicely but instruction-following on out-of-domain tasks collapses around step 800. Model starts ignoring system prompts, hallucinating JSON keys, and outputting domain-specific tokens in unrelated contexts.

context

Hardware: 4x H100. QLoRA rank=32, alpha=64, target all linear layers. LR 2e-5 cosine. Batch 128 effective. Eval on a held-out mixed benchmark (MMLU, instruction-following, tool-use).

goal

Diagnose whether this is a catastrophic forgetting problem, a data distribution problem, or an LR/rank problem. Recommend a training recipe that reaches comparable domain loss while preserving general instruction-following within 5% of base.

constraints

Keep the 70B size (we need the capability). Budget: 1 more training run on 4x H100.