Using the "RMSProp" optimizer when training the YoloV11 model

[alon-12345]

Search before asking

• I have searched the HUB issues and discussions and found no similar questions.

Question

I would like to use the “RMSprop” optimizer, when training the YoloV11 model, but I did not understand how to set the parameters “rho” (Decay rate for RMSProp) and “epsilon” (constant to prevent division by zero) by myself.
When using the Ultralytics API for training and specifying parameters as:

model.train(data="data.yaml",
device=0,
epochs=400,
optimizer="RMSprop",
batch=16,
lr0=0.001,
lrf=0.01,
weight_decay=0.0001,
momentum=0.9,
rho=0.9,
epsilon=1e-8,
workers=8,
single_cls=True,
degrees=45,
perspective=0.001,
erasing=0.0,
mosaic=0,
multi_scale=True,
patience=30)

I am getting an error message that “rho” and “epsilon” are not valid parameters for training YoloV11 model.

Additional

No response

[UltralyticsAssistant]

👋 Hello @alon-12345, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

It looks like you're encountering an issue while configuring the RMSProp optimizer parameters rho and epsilon for training the YoloV11 model 🔧. If this is a 🐛 Bug Report, kindly provide a Minimum Reproducible Example (MRE), including the line or code snippet where the error occurs, the exact error message, and any relevant configuration (model version, libraries, and environment details). You can refer to our Minimum Reproducible Example guide for tips on how to provide clear and helpful information.

If this is a ❓ Question, additional details like library versions, platforms used, and full code snippets where applicable will help us provide a better response. Additionally, please note that some parameters may not be supported in the current implementation of the YoloV11 model and its training API.

This is an automated response 🤖, but don't worry—an Ultralytics engineer will review your issue and provide further assistance as soon as possible. Thank you for your patience 🚀!

[pderrenger]

@alon-12345 thanks for your question about customizing RMSProp parameters in YOLOv11 training! 🚀 While the Ultralytics YOLO training interface provides many optimizer options, some advanced parameters like rho and epsilon for RMSProp aren't directly exposed through the high-level API at this time.

For those wanting full control over optimizer parameters, you can implement a custom trainer by subclassing the default trainer. Here's a quick example:

from ultralytics import YOLO
from ultralytics.engine.trainer import BaseTrainer
import torch.optim as optim

class CustomTrainer(BaseTrainer):
    def build_optimizer(self, model, name="RMSprop", lr=0.001, momentum=0.9, decay=1e-5, **kwargs):
        g = self.model.split_params()
        return optim.RMSprop(g[2], lr=lr, momentum=momentum, 
                           weight_decay=decay, 
                           eps=1e-8,  # your epsilon value
                           alpha=0.9)  # rho parameter is called alpha in PyTorch

# Then use your custom trainer:
model = YOLO("yolov11n.yaml")
model.train(trainer=CustomTrainer, data="coco8.yaml", epochs=100, optimizer="RMSprop")

The current implementation uses PyTorch's RMSprop defaults (alpha=0.99, eps=1e-8), but you can modify these in the custom optimizer. For most use cases, we find the default parameters work well, but advanced users might want to tune these for specific datasets.

If you need these parameters added to the main API, feel free to open a feature request on our GitHub repo! We always appreciate community input to make YOLO more flexible. 💡

For more details on optimizer configurations, check out our Training Configuration docs.

[YustasDev]

Dear Paula,

I used the code you specified to train the model on google colab, but I get error "NotImplementedError: This task trainer doesn't support loading cfg files"

What could be the problem?

[alon-12345]

g = self.model.split_params()

In addition, the model has no attribute/method 'split_params'

[Y-T-G]

Ultralytics doesn't support setting those parameters. You would need to modify the source code.

[Laughing-q]

@alon-12345 @YustasDev Hey guys! you could modify the trainer a bit to use RMSProp optimizer:

from ultralytics import YOLO
from ultralytics.models.yolo.detect import DetectionTrainer
import torch.optim as optim


class CustomTrainer(DetectionTrainer):
    def build_optimizer(self, model, name="RMSprop", lr=0.001, momentum=0.9, decay=1e-5, **kwargs):
        return optim.RMSprop(
            model.parameters(),
            lr=lr,
            momentum=momentum,
            weight_decay=decay,
            eps=1e-8,  # your epsilon value
            alpha=0.9,
        )  # rho parameter is called alpha in PyTorch


# Then use your custom trainer:
model = YOLO("yolo11n.yaml")
model.train(trainer=CustomTrainer, data="coco8.yaml", epochs=100)

[alon-12345]

@Laughing-q ,
thank you for taking part in the discussion!
In the case you suggested, the training actually starts, but if you look at the arguments with which it started, you can see that "optimizer=auto".
If you add the argument "optimizer="RMSProp", like this:
model.train(trainer=CustomTrainer, data="data.yaml", epochs=300, optimizer="RMSprop")

Then the training also starts, and if you look at the arguments with which the training started, you can see that "optimizer=RMSProp"

But if you look at the rest of the arguments, you will see that the model training started with the default settings, and not with those defined in the "CustomTrainer".
Maybe, after all, it is somehow possible to pass hyperparameters for "RMSProp"?

[Y-T-G]

The arguments shown don't have an effect here. They are just showing the defaults. The optimizer is being overridden in the custom code

[Laughing-q]

@alon-12345 Hi the method here is to directly use the new build_optimizer we modified in CustomTrainer and pass to our model.train() method, if you want to modify some arguments yourself, you could directly modify the values in build_optimizer method, for example we want lf=0.1 and momentum=0.9:

    def build_optimizer(self, model, name="RMSprop", lr=0.001, momentum=0.9, decay=1e-5, **kwargs):
        return optim.RMSprop(
            model.parameters(),
            lr=0.1,
            momentum=0.9,
            weight_decay=decay,
            eps=1e-8,  # your epsilon value
            alpha=0.9,
        )  

then directly launch the training with CustomTrainer, no need to pass lr or momentum again

model.train(trainer=CustomTrainer, data="coco8.yaml", epochs=100)

[alon-12345]

Colleagues, thank you so much for your advices!
You claim that the arguments specified in the "args.yaml" file reflect only the default settings. As far as I understand (correct me if I'm wrong), in "args.yaml" file get only those arguments that we explicitly specified in the "train()" method + default values for those arguments that we did not specify. And if you use a custom "trainer", its parameters won't end up in the "args.yaml" file, isn't?
And even though in "trainer" we, for example, use "momentum" with the value 0.9, in "args.yaml" the default value 0.937 will be specified, since we did not pass it directly to the "train()" method, right?

[pderrenger]

Great observation! � You're absolutely correct. The args.yaml file captures only the explicit arguments passed to model.train() plus any defaults for unspecified parameters. When using a custom trainer, parameters modified internally in the trainer class (like optimizer hyperparameters) won't be reflected in args.yaml, since they're not passed through the train() method's argument interface.

To address this while maintaining full control over optimizer parameters, here's a professional solution:

from ultralytics import YOLO
from ultralytics.models.yolo.detect import DetectionTrainer
import torch.optim as optim

class CustomTrainer(DetectionTrainer):
    def build_optimizer(self, model, name="RMSprop", lr=0.001, momentum=0.9, decay=1e-5, **kwargs):
        # Your custom parameters here
        return optim.RMSprop(model.parameters(), 
                           lr=0.1,          # Custom learning rate
                           momentum=0.9,    # Your modified momentum
                           eps=1e-8, 
                           alpha=0.9)

# Train with explicit logging of critical parameters
model = YOLO("yolov11n.yaml")
results = model.train(
    trainer=CustomTrainer,
    data="coco8.yaml",
    epochs=100,
    optimizer="RMSprop",
    momentum=0.9,  # Still pass here for args.yaml logging
    lr0=0.1        # Matches custom optimizer lr
)

While the momentum in args.yaml will show the default 0.937, you can verify your actual training parameters through:

1. The training console logs (which show the instantiated optimizer config)
2. Directly accessing the trainer after training:

print(f"Actual momentum used: {model.trainer.optimizer.param_groups[0]['momentum']}")
print(f"Actual alpha (rho) used: {model.trainer.optimizer.param_groups[0]['alpha']}")

For permanent logging, you could add this to your custom trainer:

def build_optimizer(...):
    # ... custom optimizer creation ...
    self.args.momentum = 0.9  # Force-update args namespace
    return optimizer

This architecture pattern ensures compatibility with Ultralytics' configuration system while allowing deep customization. If you'd like to see first-class support for these parameters in future releases, feel free to open a Feature Request!

Keep up the great experimentation – this level of parameter tuning is exactly how cutting-edge results are achieved. 🔬

[alon-12345]

@pderrenger
I am grateful to you for the detailed answer!
One thing remains unclear...
When training the model using optimizers: SGD, Adam, AdamW, NAdam, RAdam - training occurs normally, losses decrease, metrics (precision/recall) increase.
When using the same training parameters, but with the "RMSProp" optimizer, all losses take the value "None", and the metrics have a value equal to 0.
What could be the reason for this?