Reproducing problem on cityscapes

[FishYuLi]

Hello! I have read your paper very carefully and tried to reproduce your experiments on cityscapes dataset (photo to label). I changed the input size from 256 to 128 and use resnet_6blocks for G as stated in the appendix, and trained a model, and evaluated the performance of photo to label generator. You got (0.58, 0.22, 0.16) results for pixel acc, cls acc, and iou. But my best results are (0.51, 0.16, 0.10), which have quiet a large margin. I wondered if there are any other details that I should change. What is your configs for cityscapes? And can the size of batch_size produce an influence on the final results? Also, how did you evaluate the segmentation results of 128x128 generated images? Did you resize the original label image to 128x128 and do the evaluation? Thanks a lot. This work is amazing.

[junyanz]

Thanks for your nice words. We use batchSize=1 and we got worse results with a larger batchSize. I remembered a few details (e.g. saving the output as png images, loadSize=143, fineSize=128). We use the evaluation code in pix2pix. See more discussion here.

[FishYuLi]

@junyanz Ok, I see. Thanks a lot! I use batchSize=20 on 4 gpus to accelerate training. I will train the model again and try your evaluation code in pix2pix.

[FishYuLi]

@junyanz Sorry it's me again. I tried your evaluation code. It looks normal when I take the fake image as input, but it's so weird that when the input is the true image, the segmentation results become so bad. My questions are,

1. what iou does this of this caffe model get on cityscapes?
2. is there any special configs that we need to change when running normal images?
   
   The numbers are (pixel acc., cls acc., iou).
   (Do I need to new an issue in pix2pix?)

[tinghuiz]

Please see the notes from the original pix2pix torch repo (copied below):
The pre-trained model does not work well on Cityscapes in the original resolution (1024x2048) as it was trained on 256x256 images that are resized to 1024x2048. The purpose of the resizing was to 1) keep the label maps in the original high resolution untouched and 2) avoid the need of changing the standard FCN training code for Cityscapes. To get the ground-truth numbers in the paper, you need to resize the original Cityscapes images to 256x256 before running the evaluation code.

[junyanz]

Please see this discussion.

[lx7555]

I want to evaluate cityscapes datasets,because I try to reproduce the fcn score results from pip2pip by using https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix code.
I try many ways.I can not get any results when I run the scripts. All parameters are zero.
I still want to know how to configure the cityspaces dataset
There are three folders with gtFine, originals image, and predictions.
Are the gtFine and predictions color or grayscale? And what is the size of these three types of pictures?
I use python2,caffe.I Is that possible that it was because i was using python2.7 but not python3 ?
@FishYuLi

[junyanz]

@tinghuiz

[ZhangCZhen]

Hello, I evaluated the results after using the cityscape dataset, and the values obtained are far from those mentioned in the paper. So I would like to ask, is the code given by 'evaluate.py' only used to evaluate the results of label2image? Can I evaluate image2label? After I use RGB image to generate semantic segmentation results, how do I evaluate the segmentation results? Thanks a lot!

[FishYuLi]

Hello, I evaluated the results after using the cityscape dataset, and the values obtained are far from those mentioned in the paper. So I would like to ask, is the code given by 'evaluate.py' only used to evaluate the results of label2image? Can I evaluate image2label? After I use RGB image to generate semantic segmentation results, how do I evaluate the segmentation results? Thanks a lot!

@ZhangCZhen

1. Yes, the given 'evaluate.py' is for label2image. It sends the generated image to a FCN model to generate a mask, and evaluates the mask with ground truth mask.
2. Of couse you can use it to evaluate image2label. Just directly load your generated label, transfer the RGB label to H x W x C, and you can re-use the evaluation code in 'evaluate.py'.

[ZhangCZhen]

@FishYuLi Thanks for your reply.

1. When you use 'evaluate.py' to evaluate the label2RGB image results, did you get values similar to those mentioned in the paper? I used the RGB images in the dataset for testing, and the accuracy is still poor.
2. You mean I use the 'evaluate.py' to evaluate image2label results, I don't need the FCN model?

[FishYuLi]

@ZhangCZhen

1. Yes, I got almost the same FCN score.
2. Yes, FCN model is used to transfer the generated real image to label again so that we can compare it with GT masks. Please refer to the paper of Pix2pix carefully to see how exactly the evaluation metric is designed.

[ZhangCZhen]

@FishYuLi
Thank you for your reply, you are really enthusiastic！！