This project purpose is to serve itself as a concrete exercise to work on while explore segmentation in deep learning field and to be a decent workflow to prepare for more practical approaches in the future.
An implementation of various deep learning image segmentation models in digital pathology field. WSI (Whole slide Image) file is the main file we are working on by using openslide python library. We utilize openslide by dividing the pathology slides into smaller section so that we can process each section individually. After that, we can train and use models to recognize abnormal features in particular slides and finally merge the result regions together.
We decide to choose non-small cell lung cancer as targeted disease. We further divide the focus to adenocarcinoma which has 5 subtypes and 1 variant including papillary, solid, micropapillary, lepidic, acinar, and mucinous respectively.
Histopathology of Non-Small Cell Lung Cancer (Dr. Sarawut Kongkarnka)
We have limited experiences in pathology field, so advising from experts is needed and we have to narrow the targeted pattern down to lepidic, acinar, and micropapillary.
- limited computing power, especially with large files like .svs (~2GB+)
- Models' abilities to comprehend features, especially with small ones.
- Limited amount of focused cancer WSI which implies to dataset shortages
- Abnormal features that lie across multiple grid sections
- How to actually handle large output segmented images?
Commonly, in segmentation there are various deep learning models which are published with well-written documentation paper. There is existing pytorch library and tensorflow wrapper that allow us to use the predefined segmentation models at ease, both developed by qubvel teams
Each of them has different pros and cons that we need to test and compare their results and appropriately adapt to challenges.
Before we dive into possible ways to address aforementioned problem, let's list out available tools related to the problem.
| Python libs | Desktop App | Pathology tools |
|---|---|---|
| tensorflow | html css javascript | QuPath |
| keras | Nodejs | ImageJ |
| albumentations | Electronjs | |
| numpy | Openseadragon | |
| matplotlib | GeoTIFFTileSource | |
| pillow | ||
| cv2 | ||
| tifffile | ||
| openslide | ||
| pytorch |
At a glance, we can easily divide WSI file into large amount of smaller tiles. The number of tiles obviously depends on the size of extracted tiles. In this case, we choose 512px * 512px. Additionally, we need to add overlaps (~25px) between tiles to ensure that each image provides some context for certain region.
- This way models also have a chance to perceive tiny details in WSI.
- Normally, WSI dimensions are around 105px * 105px, thus, we are going to obtain large dataset from WSI.
- Along with data augmentation techniques which are processes of rotating, cropping, flipping data to gain another data out of existing data. These images will be given to models to ensure ability of generalizing images.
When the abnormal features are lying across multiple tiles, we can't use "object detection model" easily as it frames the features with boxes so it is hard to merge boxes from each tile. Instead, we use segmentation model to "color" interested features so we can easily merge grids together.
Desktop application needs to wait for the script to fully merge segmented area together on numpy memmap. We'll highlight tiles that has particular class on it on downsampled WSI. The highlighted areas will be clickable and it will take user to correspond merged result images from the models.