Faimed3d - fastai extension for volumetric medical data

Deep Learning
11

I ended up getting it to work. If anyone is using Colab they will need to:
!git clone GitHub - kbressem/faimed3d: Extension to fastai for volumetric medical data
!pip3 install torch torchvision --upgrade
!pip install -q faimed3d/

1 Like
12

@BresNet are we able to use PNGs with your library or Jpgs? I have a huge DICOM dataset that is making me run out of RAM even on Colabs Pro and I’ve made the dataset PNGs and cut the size by 70%.

13

No. In PNG/JPEG, you lose the header information from the DICOM and some voxel information due to conversion to 8bit. I would suggest you read in the DICOM and convert them to NIfTI.

dcm = TensorDicom3D.create(fn)
dcm = Resize3D((20, 224, 224))(dcm) # you can resize to e.g. 20 x 224 x 224 to save space. 
dcm.save(fn_nifiti)

Reading the NIfTI form file is also a lot faster than the DICOM series.

14

This is amazing. I am also a radiologist keen on medical imaging AI. Thank you for this contribution

1 Like
15

I am happy you like it. If you find a bug or missing features, please feel free to raise an issue.

16

@BresNet Just wondering if it is possible to use the current framework of your extension to load a second channel containing segmentation information?

So essentially where 2dCNNs use RGB channels in [R,G,B](+ positional info) per pixel, to use [intensity, segmentation mask](+positional info) per point.

I’m assuming this would require a custom dataloader to be written, but would this sort of approach work with the frame work of fasi-ai + your extension?

17

This would probably work with just fastai and the DataBlock API.
faimed3d is designed to work with volumetric images, rank 5 Tensors, and 3d CNNs. So using it for 2dCNN would just complicate things.

18

I think I have not explained myself properly sorry, let me try again.

Many Mri datasets contain multiparametric Mri scans, e.g. FLAIR, t1 t1wCE etc. What I’m wondering is if I can align these different scans, it is possible to have that data go through the 3d CNNs that are in faimed3d?

So instead of feeding in one mri type at a time, feed the different scans in at once through the 3d CNNs. The reason I mentioned 2d CNNs is that with colour images we have RGB channels.

So the question, is it possible to feed in different mri scans together at once such that each scan type, FLAIR and T1wCe, for example might go through the networks together?

In your classification example, if the 3 different axial scans were aligned and on the same plane, would you be able to feed these 3 different inputs into a CNN at once?

( I was using the example of a mask earlier but I think what I actually intend to do with having different mri types as channels makes more sense anyway)

19

Yes, that is possible. I assume, you are working on a current Kaggle challenge (RSNA MICCAI). For this challenge, I’ve already preprocessed the data. So all images have an identical orientation, spacing, and direction and are easier to read (there are still some small errors in the data, which I aim to correct soon).

The following code would probably work.


DATA_DIR = Path('/path/to/data')

df = pd.read_csv(DATA_DIR/'train_labels.csv')

# dataloaders: 
dls = ImageDataLoaders3D.from_df(df, DATA_DIR/'train', 
                                 fn_col = ['T2w', 'T1wCE', 'T1w', 'FLAIR'], 
                                 label_col = 'MGMT_value', 
                                 item_tfms = ResizeCrop3D((5, 10, 10), (40, 112, 112)), # resize
                                 bs = 8)

# model
model = create_cnn_model_3d(resnet18_3d, 
                            n_in = 4, 
                            n_out = 2, 
                            pretrained = True)

learn = Learner(dls, model, cbs = StackVolumes())

Data: RSNA MICCAI all resampled to axial view | Kaggle

2 Likes
20

Yes I was about to update my comment after I found the fn_col arg in the data.py file.

Thanks for the work, I’m finding it super helpful right now!

1 Like
21

Picking this up after a long time. Keno, I am wondering where you intend to go with this library, other than an altruistic desire to help fellow programmers? Have you any plans to collab and / or write papers based off it?

Usman

22

Hi,

my initial motivation was to implement tools for radiological images into fastai and then I decided to share it with the community, hoping it will be helpful.

I have also made some papers with the library:

However, today I mostly use MONAI and faimed3d is not actively developed anymore.
I love fastai, but for 3D medical imaging MONAI provides the better tools.

2 Likes
23

Those are great publications … Kudos!
I am keen to hear your thoughts on MONAI. I gave it a brief glance but am biased towards fastai. I have been writing some code to extend it somewhat. Have a look at my post Code collaboration opportunity for radiology projects (KiTS)

I have been able to do most things with fastai+pytorch, using torchio etc here and there, but am keen to hear where MONAI really outshines fastai.

Thanks
Usman

1 Like
24

The main reason is, that fastai is designed for 2d images and does this really well. However, 3D data, such as CT or MRI come with additional challenges. One of my main reasons to switch to MONAI were:

  1. They have 3D models, which are not provided by fastai. I have written some 3D models in faimed3d, but MONAI offers a much greater variety.
  2. CT/MRI require specific transforms, such as simulation of typical artifacts. One can use torchio but, AFAIK, torchio is not under active development anymore, instead the creator joined MONAI.
  3. MONAI provides better support of medical data and special data loaders with caching. This is important, as DICOM is often compressed, making the dataloader the bottleneck of your training.

I still use fastai for 2D images, as the training routines are better and I find it really easy to use. However, my main work nowadays is on 3D data. Similar to you, I’ve also written a training routine for segmentation :smiley: using MONAI and ignite, but its WIP GitHub - kbressem/trainlib: Template for MONAI projects

1 Like
25

Yes Monai is very very polished. I am just so in love with the fastai coding philosophy that I found it near impossible to break from it. Please keep me updated on any interesting things you find along the way :+1:

26

Any hints where I can find some work on generative models for medical images (dsa brain arteries)?

27

What do you have in mind specifically?

There is something like: https://arxiv.org/pdf/2210.04133.pdf (“Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains”)

Or something like this: Deep-Learning Generated Synthetic Double Inversion Recovery Images Improve Multiple Sclerosis Lesion Detection - PubMed

28

or this: [2209.07162] Brain Imaging Generation with Latent Diffusion Models
They even open sourced their generated data.

29

This looks great! Is it easy to adapt to support regression problems?

30

Just to clarify, I haven’t joined MONAI :smiley: