Prediction on whole slide histology images

Hi!

I am completely new to this forum so apologize in advance if someone asks something similar (didn’t find any).

I am starting to work with whole slide histology images with different classification purposes (tumor type, subtype, etc). As some of you already know, the files that you manage are extremely high resolution (sometimes 100,000x100,000 pixels and 10GB+). So, as you imagine, I am tiling the image to 224x224 size patches BUT I am kind of stucked with the labeling and/or the patch prediction aggregation. Since I do not have enough time to manually curate every patch and see if the sample have or not tumor tissue, I am more prone to use the second approach (differently from the breast dataset in kaggle). In order to do this I found this approach linked from this post that could solve my problem (the whole git is based on PyTorch, so great point here).

However, I would like to know if this could be implemented using fastai library. Any advice on this?

Thanks!
J

1 Like

Hi @imrandude,

I found your post some hours ago.

How did you implement fastai pipeline using whole slide images? Did you tile and input them as a databunch?

Thanks!
J

As an update on the issue, I found this thread in tensorflow forum. Seems that decreasing batch size to 1 and with a CPU, you could input kind of a big image without tiling.

Guess that it would take forever to train but I would give it a try. Any thoughts on this?

What is the type of tissue sample you are working with? Is your requirement to create a mask on whole slide images or any other?

Hi,

I am actually doing quite a “simple” binary classification using TCGA whole slide images from Digital Slide Archive. Concretely, I want to build a CNN that differentiates LUAD from LUSC WSI.

As you may imagine, the WSI have not only tumoral tissue, so I am not sure which is the best way to 1) tile the WSI (if necessary) 2) train 3) predict and 4) merge predictions in a meaningful way (there is a great post about this in my first comment)

Any thoughts? :slight_smile:

Just in case you missed it from the original medium post, the code below is the implementation.

Now I have not tried this with FastAI, but given that the paper has used ResNet for patch classification, we can use FastAI for much more effective training.

Yes, I end up in the post because of the paper published with this git. I guess that I could try to tile the images and use them for the training using fastai library.

However, my first post was in the direction on how to manage the prediction of the tiles in the test set in fastai. Since the WSI have non-tunoral tissue, I was wondering if fastai has some kind of function to deal with this ‘merging predictions’ issue. Do you have any idea?

Hope now is clearer what I was asking and thank you for the feedback!

Hi Joan,

Hope you have found a solution, if not you can refer to the code here (https://github.com/SBU-BMI/u24_lymphocyte) for the paper: https://arxiv.org/abs/1905.10841

Hi @imrandude,
Thanks for the paper! Actually, this is similar of what DeepSlide does but generating kind of better heatmaps for interpretation.
However, I was looking for something that unbiasly interpret this heatmaps. Imagine that, in this article, it would be a way to discern if the TILs are inside the tumor and this was a relevant clinical feature.
In my specific example, I was looking for some script that, after the predictions, postprocess them and give some new feature taking into account relative positions (which would be what the pathologist would do in the end).
I guess this is specific for every problem that you face but I was looking for some examples that I guess are not out there yet.
In any case, thank you for your help, I hope I will post some update on this at some point :smile:

1 Like

Hi @imrandude,

In this work, the authors used a RNN to solve this problem. Hope it would be helpful.

Cheers

1 Like

Thanks, are you working on implementing this with fastai?

It would be great but I am not currently doing it. Maybe in a couple of months would give it a try

Hi,
I am new too. could you please say how you tiled your WSI? I have not all of WSI, but I have some parts of slides that pathologist marked. now I need to patch images in small sizes. could you please help me?

Hi @kazemiaz,

For tiling (and training) using WSI, I guess the best starting point is start with PANDA kaggle challenge. Concretely for tiling, this notebook is a good start.

Hey everyone, I have trained my model in fastai using the MIDOG 2021 dataset, and my object detection model is working great, I have a minor issue while inferencing the model on totally new two images which are aperio images of .scn format and having aperio imagescope created annotation in XML format.
The thing is I trained my model using the fastai input pipeline based on the tutorial notebook provided by the MIDOG challenge team on their website: Google Colab
The notebook is based on the object detection library of fastai by Christian marshal:
GitHub - ChristianMarzahl/ObjectDetection: Some experiments with object detection in PyTorch
The input pipeline is as follows it takes the bunch of .tiff images and the annotations in MS COCO .json format and then creates the imagedatabunch object of fastai.
Now how should I use my new data which consists of only one .scn image and the respective . XML file of the same, which has annotations in the ellipse box (top left corner and top bottom corner coordinates) to be used for inference, and testing of my trained model.
Any resource, code snippet, notebook, etc which might be able to help to do this would be really helpful.
Thanking you all in advance,
Harshit

Hi, .SCN is an old Leica format, which is supported by OpenSlide so in principle, if the pipeline is based on it, it should be easily opened. Then Leica bought Aperio, but this is another story.
Regarding annotations, here there is some infor on conversion from ImageScope annotation (if I remember well, they are based on OME XML) to COCO:

Yes, i have successfully converted my annotation file from .XML to ms coco .json but when I try using a single image in the pipeline it gives me black and white patches in the batch, so how can I load the .scn image in fastai pipeline properly?
This is the o/p that I get after loading the image to see if it gets loaded properly or not:

image
I have cropped the image, but the bottom part of the image contains the slide and the boxes are being shown at the top most region.
I might have not done the conversion properly, I used my own code to convert . XML annotation to .json format.
Also @VDM the above tool uses qupath in between does anyone know how to properly convert the. XML aperio annotation to .json COCO file?
This is the conversion code that i used:

def to_coco(self) -> dict:
        labels = sorted(self._labels())
        label_to_id = {l: i for i, l in enumerate(labels)}
        imageid_to_id = {n: i for i, n in enumerate(self.image_ids)}
        
        annotations = []
        for annotation in tqdm(self, desc="Saving"):
            for idx, box in enumerate(annotation.boxes):
                box_annotation = {
                    #"iscrowd": 0, "ignore": 0,
                    "image_id": imageid_to_id[annotation.image_id],
                    "bbox": box.ltrb,
                    "category_id": label_to_id[box.label],
                    "id": idx}

                if box.is_detection:
                    box_annotation["score"] = box.confidence

                annotations.append(box_annotation)

        images = [{
            "id": imageid_to_id[a.image_id], 
            "file_name": a.image_id, 
            "width": a.image_width, 
            "height": a.image_height} for a in self]

        #categories = [{"supercategory": "none", "id": label_to_id[l], "name": l} for l in labels]
        categories = [{"id": label_to_id[l], "name": l} for l in labels]
        return {"images": images, "annotations": annotations, "categories": categories}

    def save_coco(self, path: Path):
        if path.suffix == "":
            path = path.with_suffix(".json")
        assert path.suffix == ".json"
        content = json.dumps(self.to_coco(), allow_nan=False)
        path.write_text(content)
    def from_aperio(
        file_path: Path, 
        image_size: "tuple[int, int]",
        image_extension: str = ".scn"
    ) -> "Annotation":
        with file_path.open() as f:
            root = et.parse(f).getroot()
        image_id = file_path.with_suffix(image_extension).name
        annotation = root.find("Annotation")
        regions = annotation.findall("Regions/Region")
        boxes = [BoundingBox.from_aperio(r) for r in regions]

        return Annotation(image_id, image_size, boxes)
    def from_aperio(node: et.Element) -> "BoundingBox":
        label = node.attrib["Text"]
        typean=node.attrib["Type"]
        if typean=="2":
            p1, p2 = node.findall("Vertices/Vertex")

            xmin, ymin = min(int(p1.attrib["X"]), int(p2.attrib["X"])), min(int(p1.attrib["Y"]), int(p2.attrib["Y"]))
            xmax, ymax = max(int(p1.attrib["X"]), int(p2.attrib["X"])),max(int(p1.attrib["Y"]), int(p2.attrib["Y"]))
        elif typean=="1":
            p1, p2, p3, p4 = node.findall("Vertices/Vertex")

            xmin, ymin = min(int(p1.attrib["X"]),int(p2.attrib["X"]),int(p3.attrib["X"]),int(p4.attrib["X"])),min(int(p1.attrib["Y"]),int(p2.attrib["Y"]),int(p3.attrib["Y"]),int(p4.attrib["Y"]))
            xmax, ymax = max(int(p1.attrib["X"]),int(p2.attrib["X"]),int(p3.attrib["X"]),int(p4.attrib["X"])),max(int(p1.attrib["Y"]),int(p2.attrib["Y"]),int(p3.attrib["Y"]),int(p4.attrib["Y"]))
        print(xmin, ymin, xmax, ymax)
        return BoundingBox(label, xmin, ymin, xmax, ymax)

Can anyone tell me how can I change the following code so that bboxes annotations are contained within the boundaries of the slide, not the whitespace padded onto image?