Document Generating Alternate Training Data / Model Weights

[jpbochicchio]

Hello,

Thank you for combining all of these different resources together into such a beautifully crafted and documented package, it is much appreciated!

One thing that I would be interested in seeing docs on is with respect to training the model for new mangas, since this one does not accurately color characters it has never seen. I used the manga colorization CycleGAN repository to train a model on colorized vs non-colorized manga pages, and actually saw some very solid results coming out of the model training itself (reference repo link below):

https://github.com/OValery16/Manga-colorization---cycle-gan/tree/master

The model itself generated numerous PTH files after training, but it looks like your model weights are saved in the generator zip as PKL files with a data folder. Any idea how to convert these sorts of PTH files to a format that your implementation will accept:

I am interested in colorizing a specific manga, so I have retrained the base CycleGAN model for this purpose alone.

Thanks for reading, and happy new year!

[jpbochicchio]

One note to add - I cloned this repo and added all of my training stuff to it for the Boruto community specifically, which can be found here if it is of interest:

https://github.com/jpbochicchio/Boruto-TwoBlueVortex-Manga-Colorizer/tree/master/Training-Utilities

Ultimately what I am looking for is documentation / information on how to properly train the colorizer model on different manga and bundle it up into the generator.zip format your project is expecting (*.pkl with a data folder rather than *.pth)

[Roncarlos]

Isn't it done by using :
https://docs.pytorch.org/docs/stable/generated/torch.save.html
https://docs.pytorch.org/docs/stable/notes/serialization.html#preserve-storage-sharing
?

[BinitDOX]

Hi!

This is the parent project for the colorizer model: https://github.com/qweasdd/manga-colorization-v2
The original training code and network aren’t provided, but it wouldn’t be difficult to recreate if needed. I’ve also trained my own model (Pix2Pix + PatchGAN), however it didn’t generalize well across different manga styles, so I’ve been using this model instead.

Since you already have a trained CycleGAN generator (.pth) and the network definition, you can simply modify colorizator.py:

• Import your network, ex: from networks.generator import CycleGenerator
• Initialize your model in the init method like: self.model = CycleGenerator().to(self.device)
• Then load the pth (pth is a much better way), model.load_state_dict(torch.load("generator.pth"))
• Can remove the tiling stuff, it was just for experiments. Wont require hints as well, can remove.
• Modify / pad / resize the image the way your model expects it. For this one it was 576px width iirc.
• In colorize(), simply run the forward pass: generated_colored = self.model(input_image)
• Post-process the output if needed, for example converting from [-1, 1] to [0, 1]:
  generated_colored = generated_colored[0].permute(1, 2, 0) * 0.5 + 0.5
  and then remove any padding, etc.

I'll look into this in 1-2days

[BinitDOX]

Load only the latest_net_G_B.pth
Seems like you need to load latest_net_G_A.pth, weird.., were the folders reversed..

[BinitDOX]

Done, please update by taking a pull.
Usage:

python inference.py [-h]
               [--input_path INPUT]
               [--output_path OUTPUT]
               [--device {cpu,cuda}]
               [--colorizer_type {AlacGAN,CycleGAN}]
               [--colorizer_path PATH]
               [--upscaler_type {ESRGAN,GigaGAN}]
               [--upscaler_path PATH]
               [--no-colorize]
               [--no-upscale]
               [--no-denoise]
               [--upscale_factor {2,4}]
               [--denoise_sigma SIGMA]

Example: python inference.py --colorizer_type=CycleGAN --upscaler_type=GigaGAN
Train your CycleGAN and replace the networks/latest_net_G_A.pth with your best latest_net_G_A.pth

And similarly for app_stream.py