TRL - Transformers Reinforcement Learning

⚡ AsyncGRPOTrainer is now stable: AsyncGRPOTrainer graduates to the stable API. It runs GRPO much faster by decoupling generation from training: a vLLM server streams rollouts while the trainer updates the model.

TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.

• Trainers: Various fine-tuning methods are easily accessible via trainers like SFTTrainer, GRPOTrainer, AsyncGRPOTrainer, DPOTrainer, KTOTrainer and more.

• Efficient and scalable:

  • Leverages 🤗 Accelerate to scale from single GPU to multi-node clusters using methods like DDP and DeepSpeed.
  • Full integration with 🤗 PEFT enables training on large models with modest hardware via quantization and LoRA/QLoRA.
  • Integrates 🦥 Unsloth for accelerating training using optimized kernels.

• Command Line Interface (CLI): A simple interface lets you fine-tune with models without needing to write code.

Install the library using pip:

pip install trl

If you want to use the latest features before an official release, you can install TRL from source:

pip install git+https://github.com/huggingface/trl.git

If you want to use the examples you can clone the repository with the following command:

git clone https://github.com/huggingface/trl.git

For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL is a light wrapper around the 🤗 Transformers trainer and natively supports distributed training methods like DDP, DeepSpeed ZeRO, and FSDP.

Here is a basic example of how to use the SFTTrainer:

from trl import SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
)
trainer.train()

GRPOTrainer implements the Group Relative Policy Optimization (GRPO) algorithm that is more memory-efficient than PPO and was used to train Deepseek AI's R1. accuracy_reward needs math-verify, which you can install with pip install trl[math_verify].

from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
)
trainer.train()

AsyncGRPOTrainer runs the same algorithm as GRPOTrainer, much faster: a vLLM server streams rollouts while the trainer updates the model, so generation and training overlap instead of alternating. The trainer needs vLLM and kernels, and accuracy_reward needs math-verify, which you can install with pip install trl[vllm,kernels,math_verify].

# train_async_grpo.py
from datasets import load_dataset
from trl import AsyncGRPOTrainer
from trl.rewards import accuracy_reward

if __name__ == "__main__":
    dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

    trainer = AsyncGRPOTrainer(
        model="Qwen/Qwen3.5-2B",
        reward_funcs=accuracy_reward,
        train_dataset=dataset,
    )
    trainer.train()

Generation runs on a vLLM server, on a different GPU than the trainer. Start it on GPU 0:

CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.5-2B \
    --weight-transfer-config '{"backend": "nccl"}' --logprobs-mode processed_logprobs --max-model-len 4096

Then run the training on GPU 1:

CUDA_VISIBLE_DEVICES=1 python train_async_grpo.py

DPOTrainer implements the popular Direct Preference Optimization (DPO) algorithm that was used to post-train Llama 3 and many other models. Here is a basic example of how to use the DPOTrainer:

from datasets import load_dataset
from trl import DPOTrainer

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
)
trainer.train()

KTOTrainer implements the Kahneman-Tversky Optimization (KTO) algorithm, which aligns models from simple binary (desirable / undesirable) feedback rather than paired preferences. Here is a basic example of how to use the KTOTrainer:

from datasets import load_dataset
from trl import KTOTrainer

dataset = load_dataset("trl-lib/kto-mix-14k", split="train")

trainer = KTOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
)
trainer.train()

Here is a basic example of how to use the RewardTrainer:

from trl import RewardTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = RewardTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=dataset,
)
trainer.train()

You can use the TRL Command Line Interface (CLI) to quickly get started with post-training methods like Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO):

SFT:

trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
    --dataset_name trl-lib/Capybara \
    --output_dir Qwen2.5-0.5B-SFT

DPO:

trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO

KTO:

trl kto --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name trl-lib/kto-mix-14k \
    --output_dir Qwen2.5-0.5B-KTO

Read more about CLI in the relevant documentation section or use --help for more details.

If you want to contribute to trl or customize it to your needs make sure to read the contribution guide and make sure you make a dev install:

git clone https://github.com/huggingface/trl.git
cd trl/
pip install -e .[dev]

A minimal incubation area is available under trl.experimental for unstable / fast-evolving features. Anything there may change or be removed in any release without notice.

Example:

from trl.experimental.new_trainer import NewTrainer

Read more in the Experimental docs.

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin and Dirhoussi, Amine and Villanova del Moral, Albert and Paniego Blanco, Sergio},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}

This repository's source code is available under the Apache-2.0 License.