Custom optimizer

[JSabadin]

Custom SGD Optimizer Proposal

This PR introduces a custom implementation of the SGD optimizer with a unique step mechanism that is crucial for optimal performance.

Here is the YAML configuration file that yields the best results using this custom training strategy:

model:
  nodes:
    - name: EfficientRep
      params:
        variant: "n"
        download_weights: False

    - name: RepPANNeck
      params:
        download_weights: False
      inputs:
        - EfficientRep

    - name: EfficientBBoxHead
      params:
        download_weights: False
      inputs:
        - RepPANNeck

  losses:
    - name: AdaptiveDetectionLoss
      attached_to: EfficientBBoxHead
      params:
        iou_type: "siou"
        n_warmup_epochs: 0
        iou_loss_weight: 20 # Should be 2.5 * accumulate_grad_batches
        class_loss_weight: 8 # Should be 1 * accumulate_grad_batches
        

  metrics:
    - name: MeanAveragePrecision
      is_main_metric: true
      attached_to: EfficientBBoxHead

  visualizers:
    - name: BBoxVisualizer
      attached_to: EfficientBBoxHead


loader:
  name: LuxonisLoaderTorch

  train_view: train
  val_view: train
  test_view: train

  params:
    dataset_name: cow_finetune
    bucket_storage: "local"

trainer:
  seed: 42
  precision: "16-mixed"
  preprocessing:
    train_image_size: [640, 640]
    keep_aspect_ratio: true
    normalize:
      active: true
      params:
        mean: [0., 0., 0.]
        std: [1, 1, 1]

  skip_last_batch: False
  batch_size: 8
  accumulate_grad_batches: 8
  epochs: &epochs 80
  num_workers: 4
  validation_interval: 10
  num_log_images: 8
  n_sanity_val_steps: 0

  training_strategy:
    name: "TripleLRSGDStrategy"
    params: 
      warmup_epochs: 3
      warmup_bias_lr: 0.1
      warmup_momentum: 0.8
      lr: 0.02
      lre: 0.0002
      momentum: 0.937     
      weight_decay: 0.0005     
      nesterov: True


  callbacks:
    - name: EMACallback
      params:
        decay: 0.9999 
        use_dynamic_decay: True 
        decay_tau: 2000
    

[codecov]

Codecov Report

Attention: Patch coverage is 64.93506% with 54 lines in your changes missing coverage. Please review.

Project coverage is 95.16%. Comparing base (631b905) to head (2951676).
Report is 16 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
luxonis_train/strategies/triple_lr_sgd.py	30.15%	44 Missing ⚠️
luxonis_train/models/luxonis_lightning.py	50.00%	6 Missing ⚠️
luxonis_train/callbacks/training_manager.py	62.50%	3 Missing ⚠️
luxonis_train/strategies/base_strategy.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #132      +/-   ##
==========================================
- Coverage   96.31%   95.16%   -1.16%     
==========================================
  Files         147      164      +17     
  Lines        6304     7160     +856     
==========================================
+ Hits         6072     6814     +742     
- Misses        232      346     +114     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.