Deep learning with medical images


I was listening to @jeremy on TWIML last week, and at the end of the talk he mentions the lack of publicly available medical imaging datasets. Applying deep learning to medical images is my research area, so I am intimately aware of the problem and thought I could contribute what I’ve learned about the practice back to the community.

I created a blog post with my thoughts on how to get started with using deep learning on medical images, specifically magnetic resonance (MR) and computed tomography (CT) images. I overview the two imaging modalities, suggest several publicly available datasets, discuss some techniques for data wrangling and preprocessing (with example scripts), and finally build a small 3D deep learning model using the fastai API.

It turned out a bit longer than expected, and while there is a lot more information to cover, this should (hopefully) help people get started with applying deep learning to structural MR and CT images. I know there has been some previous discussion on here (see here, here, and here for some previous discussion). But I’d be happy to answer any questions regarding the blog post or more general questions regarding working with medical images.

Just wanted to say thanks to everyone who has helped build the fastai package, it’s awesome!


Hi there! Good write-up :slight_smile:

I am one of the developers of NiftyNet, which you mention in the post: I would just like to add that we’ve put out a demo of using NiftyNet image readers/writers within PyTorch.

It shows how to get medical data into the PyTorch context and also how to output results in the correct format. As you mentioned, these operations are different in medical imaging compared to the normal computer-vision based approach, and we think our library makes this part of it much easier.

I’m also keen to talk more generally about medical imaging and will be following this thread eagerly.

(Angel Isaac Antonio Brisco) #3

Hi I’m starting a project for my tesis about pancreas segmentation for pancreatic cancer diagnosis. I’ve been reading some articles about it, but as you say there are few databases for practicing and fewer articles that descrive with detail their neuralnet architecture.
Thanks for the blog posts and for the effort of including examples.
I have a question, how do you deal with the absence of context in the 2d slice of CT VS a 3d analysis?