Skip to content

Fix model #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: v1.1.0
Choose a base branch
from
Open

Fix model #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
00123e0
fix config and fluid api.
qizhaoaoe Mar 24, 2023
c8d3e8c
fix simclr config.
qizhaoaoe Mar 26, 2023
8f2304e
fix byol lr scheduler with warmup and global_batch_size setting in tr…
qizhaoaoe Mar 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions configs/byol/byol_r50_IM.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
epochs: 300
use_byol_iters: True
use_simclr_iters: True
total_images: 1281167
global_batch_size: 4096
output_dir: output_dir
Expand Down Expand Up @@ -84,12 +84,13 @@ dataloader:


lr_scheduler:
name: CosineWarmup
learning_rate: 4.8
T_max: 93835
warmup_steps: 3127
start_lr: 0.0001
end_lr: 4.8
name: simclrCosineWarmup
learning_rate_scaling: linear
total_images: 1281167
warmup_epochs: 10
start_lr: 0
end_lr: 0.3
T_max: 300


optimizer:
Expand Down
17 changes: 11 additions & 6 deletions configs/simclr/simclr_r50_IM.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ epochs: 100
use_simclr_iters: True
global_batch_size: 4096
output_dir: output_dir
device: gpu

model:
name: SimCLR
Expand All @@ -21,7 +22,9 @@ model:

dataloader:
train:
num_workers: 6
loader:
num_workers: 4
use_shared_memory: True
sampler:
batch_size: 32
shuffle: true
Expand Down Expand Up @@ -83,9 +86,11 @@ dataloader:
std: [0.229, 0.224, 0.225]

val:
num_workers: 4
loader:
num_workers: 4
use_shared_memory: True
sampler:
batch_size: 512
batch_size: 256
shuffle: false
drop_last: false
dataset:
Expand All @@ -105,18 +110,18 @@ dataloader:

lr_scheduler:
name: simclrCosineWarmup
learning_rate_scaling: sqrt
learning_rate_scaling: linear
total_images: 1281167
warmup_epochs: 10
start_lr: 0
end_lr: 1.0
end_lr: 0.3
T_max: 200


optimizer:
name: LarsMomentumOptimizer
momentum: 0.9
lars_weight_decay: 0.0001
lars_weight_decay: 1e-6
exclude_from_weight_decay: ["scale","offset",".bias"]

log_config:
Expand Down
12 changes: 4 additions & 8 deletions passl/engine/trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,21 +145,17 @@ def __init__(self, cfg):
self.train_dataloader, self.mixup_fn = build_dataloader(
cfg.dataloader.train, self.device)
self.iters_per_epoch = len(self.train_dataloader)

self.batch_size = cfg.dataloader.train.sampler.batch_size
self.global_batch_size = self.batch_size * dist.get_world_size()
# use byol iters
if self.use_byol_iters:
self.global_batch_size = cfg.global_batch_size
self.byol_total_iters = self.epochs * cfg.total_images // self.global_batch_size

if self.use_byol_iters:
self.lr_scheduler = build_lr_scheduler(cfg.lr_scheduler,
self.byol_total_iters)
elif self.use_simclr_iters:
self.batch_size = cfg.dataloader.train.sampler.batch_size
self.global_batch_size = cfg.global_batch_size
self.epochs = cfg.epochs
self.lr_scheduler = build_lr_scheduler_simclr(
cfg.lr_scheduler, self.iters_per_epoch, self.batch_size * 8,
cfg.lr_scheduler, self.iters_per_epoch, self.global_batch_size,
cfg.epochs, self.current_iter)
else:
self.lr_scheduler = build_lr_scheduler(cfg.lr_scheduler,
Expand Down Expand Up @@ -224,7 +220,7 @@ def __init__(self, cfg):
self.add_train_hooks()
self.add_custom_hooks()
self.hooks = sorted(self.hooks, key=lambda x: x.priority)

print("hooks: ", self.hooks)
if self.epochs:
self.total_iters = self.epochs * self.iters_per_epoch
self.by_epoch = True
Expand Down
Loading