Iām going to edit the code above shortly with some changes I made.
In MRI, a slice is a single image in a stack. You can use a 2D convnet to perform image-level classification, then derive study-level classification from that. Or you can use a 3D convnet to achieve study-level classification by treating the entire stack of images as a volume. There are also some hybrid approaches using object detection architectures, like this paper.
Awesome work guys!
awesome, thanks for the notebook! Nice to be able to see the data now.
Do you guys have any preference on an eventual virtual meetup. We could perhaps do a google hangouts. What timezones are people here in? (me GMT +2)
Hi. Thanks to @melonkernel for creating this thread. I just read all posts and saw your proposals of models and links to papers. Great 
However, why not starting with the model of the original paper āDeep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNetā:
Code for replicating these findings is provided as Supporting Information (S1 Code and S2 Code).
At first, we could turn this code into fastai in order to get the same results (even exceed them thanks to the great ideas of fastai 
Thus, we could post in github the fastai notebook and put the weights online (in order to help the DL and health community not to train the same models again and again from scratch).
Then, as a team or through different teams (here at Brasilia, we are creating a team for this competition), we could train better models to improve our results.
What do you think of that?
The solution is done in another thread about MRNet : Stanford MRNet Competition trouble downloading dataset to gcp (I guess it is what @nswitanek used, no?):
wget ...) the 6 Go MRNet dataset: it works well
The structure after unzip is the following one:
- train
-- axial
-- coronal
-- sagittal
- train-abnormal.csv
- train-acl.csv
- train-meniscus.csv
- valid
-- axial
-- coronal
-- sagittal
- valid-abnormal.csv
- valid-acl.csv
- valid-meniscus.csv
With the command du -h MRNet-v1.0 in my terminal ubuntu, I get the following folder sizes:
258M MRNet-v1.0/valid/axial
222M MRNet-v1.0/valid/coronal
230M MRNet-v1.0/valid/sagittal
709M MRNet-v1.0/valid
2.4G MRNet-v1.0/train/axial
2.1G MRNet-v1.0/train/coronal
2.2G MRNet-v1.0/train/sagittal
6.6G MRNet-v1.0/train
7.3G MRNet-v1.0Hi Pierre,
Thanks - my issue was on Windows but yes I may have to download to a server as no luck yet on Windows!
I have been able to download and run the above code to view the images. But I am using mac.
@LessW2020 @rsrivastava : in fact, I did not try to download the MRNet dataset to a mac or a windows notebook but to my instance in GPC. I guess that the Google Extension given works in all situations.
It is working for me too.
I just submitted a PR to the repo with my EDA nb.
Iāve edited my code above with a little refactoring/upgrading of the original KneePlot class. But the big addition is a new class that generates an interactive plot of all three imaging planes (axial, sagittal and coronal) for each case.
This is the code to load the data for use in each class.
train_abnl = pd.read_csv(data_path/'train-abnormal.csv', header=None,
names=['Case', 'Abnormal'],
dtype={'Case': str, 'Abnormal': np.int64})
def load_one_stack(case, data_path=train_path, plane='coronal'):
fpath = data_path/plane/'{}.npy'.format(case)
return np.load(fpath)
def load_stacks(case):
x = {}
planes = ['axial', 'coronal', 'sagittal']
for i, plane in enumerate(planes):
x[plane] = load_one_stack(case, plane=plane)
return x
case = train_abnl.Case[0]
x = load_one_stack(case)
x_multi = load_stacks(case)
And this is the code for the new class to generate the multi-viewer.
from ipywidgets import interact, Dropdown, IntSlider
class MultiKneePlot():
def __init__(self, x_multi, figsize=(10, 10)):
self.x = x_multi
self.planes = ['coronal', 'sagittal', 'axial']
self.slice_nums = {plane: self.x[plane].shape[0] for plane in self.planes}
self.figsize = figsize
def _plot_slices(self, plane, im_slice):
fig, ax = plt.subplots(1, 1, figsize=self.figsize)
ax.imshow(self.x[plane][im_slice, :, :])
plt.show()
def draw(self):
planes_widget = Dropdown(options=self.planes)
plane_init = self.planes[0]
slice_init = self.slice_nums[plane_init] - 1
slices_widget = IntSlider(min=0, max=slice_init, value=slice_init//2)
def update_slices_widget(*args):
slices_widget.max = self.slice_nums[planes_widget.value] - 1
slices_widget.value = slices_widget.max // 2
planes_widget.observe(update_slices_widget, 'value')
interact(self._plot_slices, plane=planes_widget, im_slice=slices_widget)
def resize(self, figsize): self.figsize = figsize
Example usage with screenshot of viewer.
plot_multi = MultiKneePlot(x_multi)
plot_multi.draw()
@neuradai Thanks a ton for the notebook - itās merged into the master so everyone can use.
We got our resnet (XResNet!) tonight as I had expected/hoped, along with the final updates to FastAI 1.2 so I think we are set in terms of framework and base CNN to work with.
I did do some testing with the Lisht activation and it looks promising, though itās prone to exploding due to the fast learning rateā¦but I will try and test with it in the XResNet soon.
I see we can do polls here, so maybe we need to do that but imo, the current question is how do we want to get started?
One idea I had was to just build a tiny toy dataset with the images and throw it into XResNet as a simple test to start seeing how that performsā¦then build up from there?
Iām in US PST and can do a video chat/hangout at pretty flex times.
@pierreguillou - thanks for the link to the Stanford code. I did not know it was publishedā¦I downloaded itā¦
and I had to double check b/c the model is is so basic. Here it is:
import torch
import torch.nn as nnfrom torchvision import models
class MRNet(nn.Module):
def init(self):
super().init()
self.model = models.alexnet(pretrained=True)
self.gap = nn.AdaptiveAvgPool2d(1)
self.classifier = nn.Linear(256, 1)def forward(self, x): x = torch.squeeze(x, dim=0) # only batch size 1 supported x = self.model.features(x) x = self.gap(x).view(x.size(0), -1) x = torch.max(x, 0, keepdim=True)[0] x = self.classifier(x) return x
I still question if I missed something b/c this looks like something an intern built, but anyway, if thatās their model then we should be able to blow the doors off of them with FastAI
I propose creating a toy dataset by creating a subset of the data balanced by āabnormalā positive/negative. Then take the middle slice from each plane to create a 3-channel stack for input into the XResNet.
train_data = np.zeros(train_df_reduced.shape[0], 3, 256, 256)
planes = [āaxialā, āsagittalā, ācoronalā]
for i, case in train_df_reduced.Case.iteritems():
for j, plane in enumerate(planes):
data = np.load(ā{}/{}.npyā.format(plane, case)
mid_slice = data.shape[0] // 2
train_data[i, j, :, :] = data[mid_slice, :, :]
Thereās surely a more fast.ai-y way to do this, but Iām just trying to illustrate my suggestion here with code.
Iām in US PST and same for @rsrivastava.
I think a video chat would be great, but at the same time, we may be able to just work via these forums here and possibly some kind of slack channel or similar.
It may be too difficult to get everyone online at the same time in general?
Iāll definitely vote for this toy dataset proposalā¦I think the faster we get to actually running some scans through XResNet and get some vague idea of how it does, the better.
That way we can start getting actual feedback asap on what XResNet can and canāt readily pickup on, and then we start looking at more complicated architectures/inputs, testing more images as input, testing whether we need segmentation and/or super resolution,etc.
It will be interesting also to try a small mixup test and see if that does any good on this starter set.
Iām going to try and play with the new XResNet via the notebook later today if possible.
I still canāt get the dataset to unzip hereā¦Iāll reboot later today and try again, but if you make a basic starter dataset, can you post it somewhere or push it to the repro, so I can download and test with it?
Yes, I used wget to download it, to GCP.
Discourse has been warning me that Iāve been posting too much 
, so Iāve decided to use a Medium post as a sort of ālab notebookā for my data exploration (with included domain knowledge).
Update: My post was published by Towards Data Science. Iāve changed the link below to the friends link, so FastAI folks can bypass the paywall, if needed. Please donāt share this link beyond these forums, though.
For discussion of model architectures and the like, Iāll keep most of my responses here, rather than include that info in the post. Donāt want to give any of our competitors too much of an advantageā¦
Thanks, @neuradai.
Iām extending your EDA nb a little. Working from a subsample I see that the number of images per sequence varies across cases:
Is such variation expected? Why does it occur? Is the middle slice guaranteed to be centered in the same place across patients?
Note that there are five distinct classes of cases:
The cases that are considered Abnormal but are without either ACL or Meniscus tear are the most common category. ACL tears without a Meniscus tear is the least commonly occurring condition in the training sample.
That is expected, due to differences in patient size and the orientation of their leg during the scan. If youāre processing a batch of full sequences, youāll need to pad the sequences for each plane with 256 x 256 zero arrays - probably on either side.
There are no guarantees in medical imaging, so the mid-slice is not guaranteed to be centered on the knee joint in all planes for all patients. However, it is standard practice for the MR technologist to attempt this when acquiring the images in the coronal and sagittal planes. Furthermore, given the location of the structures of interest (ACL, medial and lateral menisci), itās not guaranteed that weāll see all of the structures in any one slice - let alone the middle slice for that sequence.
Also, note the ābadā data points section of my Medium post, where there are some images that were either poorly cropped in the preprocessing by the Stanford ML Group or technologist error in acquiring them.
Iāve added this bit of code to look at the minimum number of slices (centered around the middle slice) needed to include all of the relevant structures. In my limited exploration thus far, the answer is ~ 16 - with the exception of ābadā data points. Conveniently, this would also keep stack sizes the same for all planes, if the stats for minimum slices hold from your screenshot, when applied to the entire data set.
def load_partial_stacks(case, data_path=train_path, slice_limit=None):
x = {}
planes = ['coronal', 'sagittal', 'axial']
if not slice_limit:
return load_stacks(case, data_path)
else:
for i, plane in enumerate(planes):
data = load_one_stack(case, data_path, plane)
if slice_limit >= data.shape[0]:
x[plane] = data
else:
mid_slice = data.shape[0] // 2
lower = mid_slice - (slice_limit // 2)
upper = mid_slice + (slice_limit // 2)
x[plane] = data[lower:upper, :, :]
return x