POLICY · MODELS · #2285
GPT-OSS-20B fine-tuned for multilingual reasoning on Amazon SageMaker HyperPod
Teams fine-tuned the open-weight reasoning model GPT-OSS-20B on a multilingual chain-of-thought dataset (HuggingFaceH4/Multilingual-Thinking) using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) on Amazon SageMaker HyperPod with EKS, LoRA, TRL/veRL, vLLM, Ray/KubeRay and Amazon FSx for Lustre. Results: the base model almost always produced English chains-of-thought; SFT corrected most of the language behavior gap, and GRPO provided additional improvements only when the desired behavior had non-zero probability in the base model; a Qwen2.5-7B control experiment supported the same conclusion.
KEY POINTS
- Teams fine-tuned the open-weight reasoning model GPT-OSS-20B on a multilingual chain-of-thought dataset (HuggingFaceH4/Multilingual-Thinking) using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) on Amazon SageMaker HyperPod with EKS, LoRA, TRL/veRL, vLLM, Ray/KubeRay and Amazon FSx for Lustre.
- Results: the base model almost always produced English chains-of-thought; SFT corrected most of the language behavior gap, and GRPO provided additional improvements only when the desired behavior had non-zero probability in the base model; a Qwen2.5-7B control experiment supported the same conclusion.
- This shows practical guidance: SFT typically teaches behaviors absent from an open-weight model, while GRPO refines behaviors that already appear with some probability, and demonstrates a scalable SageMaker HyperPod stack for such fine-tuning.
WHY IT MATTERS
This shows practical guidance: SFT typically teaches behaviors absent from an open-weight model, while GRPO refines behaviors that already appear with some probability, and demonstrates a scalable SageMaker HyperPod stack for such fine-tuning.