Further training your model on new images

[LimnosTiger]

Hi All,

Does anyone know how to put new images into their training data set so you can train your model further?

I attempted this by pasting my new images into the training folder I believed my model was pulling from and then pressing 'extend dataset' from the extras menu.

I'm worried about deleting some the older images incase it messes up the model or crashes. I had this occur with the previous model when I accidently changed the name of the training images folder. Even though I changed the name back it would not reopen the project and begin training again.

If I create a new project and copy and paste the model from the previous project into the model folder, will the model work the same?

Is it possible to change it so rootpainter picks images from a new specific folder instead of randomly picking images from the old training image folder which had 600 images and there is only a slim chance that my 50 new images will be called apon to annotate?

Thanks!

LT

[Abe404]

Hi LT,

The way the extend dataset function works is that it will re-read the dataset folder and check for new images that are not already referenced in your project file. If it finds new images that are not already in the project file then it will extend the project file with these new images.

The project file defines which images from the dataset are referenced by the RootPainter project and in which order they should appear in the graphical user interface (The RootPainter client).

In your situation, I understand you both want to 'trim' the existing project i.e remove some of your existing images from the project that were already loaded into the initial project file and then subsequently extend the project dataset with the new images.

I suggest you open the project file in a text editor so you can see what is going on. It's fairly straightforward to edit it yourself, but be mindful of deleting entries for which you already have annotations, or you will never be able to go back and edit these annotations in the client.

Here's an example scenario:

1. Create project with 600 images in training dataset.
2. Annotate 200 images and decide the remaining 400 are no longer of interest for you to annotate or train on.
3. To solve this you open the project file and delete the remaining 400 images.
4. Then you delete these remaining 400 images from your training dataset folder.
5. You add 200 of your new datasets images to your training dataset folder and 'extend project' using RootPainter to add these into your current project.

The hard part of the above is step 4: Deleting images from a folder that you deleted from the project file, as you'd need to write a python script or similar to make sure you deleted the right images.

Another solution:

1. Create project with 600 images in training dataset.
2. Annotate 200 images and decide the remaining 400 are no longer of interest for you to annotate or train on.
3. You add 200 images from your new dataset to your training dataset folder.
4. You run 'extend dataset' with RootPainter. You now have 800 images in your project but you want to navigate to the last 200 as you want to focus the model development/training on these.
5. You trigger the RootPainter client to 'skip' from image 200 to image 600 by editing the project file yourself. You basically open the project file and just delete entries from 200 to 600, so when you re-open the project is will only have 400 entries. 200 from the original dataset and 200 from your new dataset.

With this second solution, your dataset folder still contains some forever 'unused' images from your original dataset, but this doesn't matter because they aren't referenced in the project file so will never be annotated, which also means they will never be used in training.

The hardest part is opening the project file and editing it. But this is just a text file - and it seems like you want some extra control, so perhaps familiarising yourself with this text file will be worthwhile.

Kind regards,
Abraham

[LimnosTiger]

Hi Abraham,

Thank you very much for your detailed response. Really appreciate it, and thank you for making such a fantastic tool!

You've understood the problem I have very well. That looks like a great fix. I'll be sure to try it shortly.

I've been using RootPainter on minirhizotron images in wheat over the course of the season. The idea was instead of creating an entirely new model, to annotate an additional 20-30 difficult images as soil conditions and root structures change slightly over time.

Hopefully, I can get your solution to work. Following your instructions, it does not look too difficult and potentially worthwhile, as you suggested, becoming more familiar with the system. I'll let you know how I go.

Kind regards,

LT

[Abe404]

Good luck. You may be interested in the following study:
https://acsess.onlinelibrary.wiley.com/doi/full/10.1002/ppj2.20094