Loss does not decrease accordingly

[mertmerci]

I'm running the training file using the Cityscapes datasets mentioned, however the loss does not decrease accordingly. In the beginningg it was around 2, in 90th epoch it's struggling around 1. All my hyper parameters are at default; the learning rate is 1e-2, momentum is 0.9 and weight decay is 1e-4.

[Tramac]

Hi, maybe your lr is too large, try 0.001 instead.

[CuttlefishXuan]

@mertmerci Hi, I have the same situation as you. And I tried lr=0.001 or lr=0.0003, it still doesn't work. Have you solved this problem? what't your best training loss you can get now?

[mertmerci]

I have not solved the problem yet. The best loss I can get is around 0.7.

[Linda-L]

Epoch: [ 5/500] Iter [ 345/ 647] || Time: 1166.8377 sec || lr: 0.00990037 || Loss: 0.6414
Epoch: [ 5/500] Iter [ 355/ 647] || Time: 1170.0967 sec || lr: 0.00990010 || Loss: 0.4728
Epoch: [ 5/500] Iter [ 365/ 647] || Time: 1173.3334 sec || lr: 0.00989982 || Loss: 0.6681
Epoch: [ 5/500] Iter [ 375/ 647] || Time: 1176.5961 sec || lr: 0.00989954 || Loss: 0.5141
Epoch: [ 5/500] Iter [ 385/ 647] || Time: 1179.8637 sec || lr: 0.00989926 || Loss: 0.4688
Epoch: [ 5/500] Iter [ 395/ 647] || Time: 1183.0879 sec || lr: 0.00989898 || Loss: 0.6347
Epoch: [ 5/500] Iter [ 405/ 647] || Time: 1186.2374 sec || lr: 0.00989870 || Loss: 0.4698
Epoch: [ 5/500] Iter [ 415/ 647] || Time: 1189.4530 sec || lr: 0.00989842 || Loss: 0.6898
Epoch: [ 5/500] Iter [ 425/ 647] || Time: 1192.6652 sec || lr: 0.00989815 || Loss: 0.5331
Epoch: [ 5/500] Iter [ 435/ 647] || Time: 1195.9574 sec || lr: 0.00989787 || Loss: 0.7055
Epoch: [ 5/500] Iter [ 445/ 647] || Time: 1199.3273 sec || lr: 0.00989759 || Loss: 0.5093
Epoch: [ 5/500] Iter [ 455/ 647] || Time: 1202.6462 sec || lr: 0.00989731 || Loss: 0.4337
Epoch: [ 5/500] Iter [ 465/ 647] || Time: 1205.8542 sec || lr: 0.00989703 || Loss: 0.4287
Epoch: [ 5/500] Iter [ 475/ 647] || Time: 1209.0492 sec || lr: 0.00989675 || Loss: 0.6152
Epoch: [ 5/500] Iter [ 485/ 647] || Time: 1212.3493 sec || lr: 0.00989647 || Loss: 1.2922
Epoch: [ 5/500] Iter [ 495/ 647] || Time: 1215.4831 sec || lr: 0.00989620 || Loss: 0.4719
Epoch: [ 5/500] Iter [ 505/ 647] || Time: 1218.5399 sec || lr: 0.00989592 || Loss: 0.9275
Epoch: [ 5/500] Iter [ 515/ 647] || Time: 1221.7445 sec || lr: 0.00989564 || Loss: 0.5310
Epoch: [ 5/500] Iter [ 525/ 647] || Time: 1224.8567 sec || lr: 0.00989536 || Loss: 0.7151
Epoch: [ 5/500] Iter [ 535/ 647] || Time: 1228.1229 sec || lr: 0.00989508 || Loss: 0.4688
Epoch: [ 5/500] Iter [ 545/ 647] || Time: 1231.3132 sec || lr: 0.00989480 || Loss: 0.5770
Epoch: [ 5/500] Iter [ 555/ 647] || Time: 1234.6185 sec || lr: 0.00989453 || Loss: 0.6590
Epoch: [ 5/500] Iter [ 565/ 647] || Time: 1238.0210 sec || lr: 0.00989425 || Loss: 0.3722

[Linda-L]

Epoch: [12/500] Iter [ 166/ 647] || Time: 2583.1239 sec || lr: 0.00977914 || Loss: 0.5476
Epoch: [12/500] Iter [ 176/ 647] || Time: 2586.6632 sec || lr: 0.00977886 || Loss: 0.7562
Epoch: [12/500] Iter [ 186/ 647] || Time: 2590.0186 sec || lr: 0.00977858 || Loss: 0.5186
Epoch: [12/500] Iter [ 196/ 647] || Time: 2593.0659 sec || lr: 0.00977830 || Loss: 1.0565
Epoch: [12/500] Iter [ 206/ 647] || Time: 2596.3114 sec || lr: 0.00977802 || Loss: 0.6302
Epoch: [12/500] Iter [ 216/ 647] || Time: 2599.6234 sec || lr: 0.00977774 || Loss: 0.5147
Epoch: [12/500] Iter [ 226/ 647] || Time: 2602.8056 sec || lr: 0.00977746 || Loss: 0.8301
Epoch: [12/500] Iter [ 236/ 647] || Time: 2605.9763 sec || lr: 0.00977718 || Loss: 0.5063
Epoch: [12/500] Iter [ 246/ 647] || Time: 2609.2808 sec || lr: 0.00977690 || Loss: 0.6625
Epoch: [12/500] Iter [ 256/ 647] || Time: 2612.6594 sec || lr: 0.00977663 || Loss: 0.6614
Epoch: [12/500] Iter [ 266/ 647] || Time: 2615.8317 sec || lr: 0.00977635 || Loss: 1.1284
Epoch: [12/500] Iter [ 276/ 647] || Time: 2619.1268 sec || lr: 0.00977607 || Loss: 0.6293
Epoch: [12/500] Iter [ 286/ 647] || Time: 2622.3897 sec || lr: 0.00977579 || Loss: 0.4887
Epoch: [12/500] Iter [ 296/ 647] || Time: 2625.7109 sec || lr: 0.00977551 || Loss: 0.6670
Epoch: [12/500] Iter [ 306/ 647] || Time: 2629.1389 sec || lr: 0.00977523 || Loss: 0.6722

Train my own data set, there are 2 categories (including background)

[Tramac]

Epoch: [12/500] Iter [ 166/ 647] || Time: 2583.1239 sec || lr: 0.00977914 || Loss: 0.5476
Epoch: [12/500] Iter [ 176/ 647] || Time: 2586.6632 sec || lr: 0.00977886 || Loss: 0.7562
Epoch: [12/500] Iter [ 186/ 647] || Time: 2590.0186 sec || lr: 0.00977858 || Loss: 0.5186
Epoch: [12/500] Iter [ 196/ 647] || Time: 2593.0659 sec || lr: 0.00977830 || Loss: 1.0565
Epoch: [12/500] Iter [ 206/ 647] || Time: 2596.3114 sec || lr: 0.00977802 || Loss: 0.6302
Epoch: [12/500] Iter [ 216/ 647] || Time: 2599.6234 sec || lr: 0.00977774 || Loss: 0.5147
Epoch: [12/500] Iter [ 226/ 647] || Time: 2602.8056 sec || lr: 0.00977746 || Loss: 0.8301
Epoch: [12/500] Iter [ 236/ 647] || Time: 2605.9763 sec || lr: 0.00977718 || Loss: 0.5063
Epoch: [12/500] Iter [ 246/ 647] || Time: 2609.2808 sec || lr: 0.00977690 || Loss: 0.6625
Epoch: [12/500] Iter [ 256/ 647] || Time: 2612.6594 sec || lr: 0.00977663 || Loss: 0.6614
Epoch: [12/500] Iter [ 266/ 647] || Time: 2615.8317 sec || lr: 0.00977635 || Loss: 1.1284
Epoch: [12/500] Iter [ 276/ 647] || Time: 2619.1268 sec || lr: 0.00977607 || Loss: 0.6293
Epoch: [12/500] Iter [ 286/ 647] || Time: 2622.3897 sec || lr: 0.00977579 || Loss: 0.4887
Epoch: [12/500] Iter [ 296/ 647] || Time: 2625.7109 sec || lr: 0.00977551 || Loss: 0.6670
Epoch: [12/500] Iter [ 306/ 647] || Time: 2629.1389 sec || lr: 0.00977523 || Loss: 0.6722

Train my own data set, there are 2 categories (including background)

Please confirm that the label is correct, maybe we can talk through the WeChat more smoothly.

[Linda-L]

Epoch: [12/500] Iter [ 166/ 647] || Time: 2583.1239 sec || lr: 0.00977914 || Loss: 0.5476
Epoch: [12/500] Iter [ 176/ 647] || Time: 2586.6632 sec || lr: 0.00977886 || Loss: 0.7562
Epoch: [12/500] Iter [ 186/ 647] || Time: 2590.0186 sec || lr: 0.00977858 || Loss: 0.5186
Epoch: [12/500] Iter [ 196/ 647] || Time: 2593.0659 sec || lr: 0.00977830 || Loss: 1.0565
Epoch: [12/500] Iter [ 206/ 647] || Time: 2596.3114 sec || lr: 0.00977802 || Loss: 0.6302
Epoch: [12/500] Iter [ 216/ 647] || Time: 2599.6234 sec || lr: 0.00977774 || Loss: 0.5147
Epoch: [12/500] Iter [ 226/ 647] || Time: 2602.8056 sec || lr: 0.00977746 || Loss: 0.8301
Epoch: [12/500] Iter [ 236/ 647] || Time: 2605.9763 sec || lr: 0.00977718 || Loss: 0.5063
Epoch: [12/500] Iter [ 246/ 647] || Time: 2609.2808 sec || lr: 0.00977690 || Loss: 0.6625
Epoch: [12/500] Iter [ 256/ 647] || Time: 2612.6594 sec || lr: 0.00977663 || Loss: 0.6614
Epoch: [12/500] Iter [ 266/ 647] || Time: 2615.8317 sec || lr: 0.00977635 || Loss: 1.1284
Epoch: [12/500] Iter [ 276/ 647] || Time: 2619.1268 sec || lr: 0.00977607 || Loss: 0.6293
Epoch: [12/500] Iter [ 286/ 647] || Time: 2622.3897 sec || lr: 0.00977579 || Loss: 0.4887
Epoch: [12/500] Iter [ 296/ 647] || Time: 2625.7109 sec || lr: 0.00977551 || Loss: 0.6670
Epoch: [12/500] Iter [ 306/ 647] || Time: 2629.1389 sec || lr: 0.00977523 || Loss: 0.6722
Train my own data set, there are 2 categories (including background)

Please confirm that the label is correct, maybe we can talk through the WeChat more smoothly.

Thank you very much , and this is my wechat phone number,13716670388

[juliuswunderlich]

Having the same problem. What was the solution?

[Alohomora0130]

I have the same problem. Have you solve this problem? Could you share us with your learning rate and batch_size? Thank you!