How many images did you use for training when you only had the 63 images? And how many images did you use for training when you had the full 108 images? Are you testing on the same validation set for both cases?
With the 63 patients I used: 1460, 353 and 323 2D slices for training, validation and test sets respectively and with 100 (I am actually using only 100 not 108, sorry) I used: 4035, 520 and 431 respectively. in the first case I split data in 70/15/15 percent and in the second 80/10/10. I recently tried to use the same validation and test set with the 100 patients that I used with the 63, the performance in the validation set was 70% F1score but in the test set was good like with 63 patients, but it seems tricky, that make me think that maybe the test set is special in some way, because when I shuffle the patients in a different order the results are not as good.
So with the 63 patients there are 2136 slices, but adding 37 more patients results in 4986 slices? That would suggest to me that those extra images are very different than the original 63 data sets. I would examine both sets of data and make sure that the 2D slices are consistent and the dataāqualitativelyālooks similar. My impression is that this disparity in performance is a result of poor data harmonization.
1 Like
Have you ensured that slices from the same patient do not end up in test, validation and train, if this has happened it can influence the results and make the validation look artificially good (assuming adjacent slices can be in some respects similar)
2 Likes
Yes, I was going to mention that the typical way to split the data is by patient. This really is necessary to judge the performance in a meaningful capacity.
Yes, there are not slices from the same patient in any of the subsets, I realized that in my first try, I had not optimized anything and I already had a good performance, but yes i have been careful about that, first I split the patients and then I extract the slices.
1 Like
You have a point, I was wrong in the number of images, by adding 37 images I have 3,481 in total, any way I will try to evaluate the harmony of the images and add some standardization. On the other hand, I have another question, what do you think is a good number of images for a binary classification using 3D convnets? Do you think that with 100 or even 250 patients I could have a good performance?
Thank you for the advice,
The performance will depend on the application. Are you planning on using the model with data from a site/scanner not seen in the training set?
For testing your model, Iād randomly shuffle the images and take 80 for training, 10 for validation, and 10 for testing.
Yes, we are planning to test the model in a different data set not seen in the training set, there is another publicly available data set and we have our own data set but this is a small one around 15 patients.
Just to be sure, for testing my model you mean my actual model with 2D convnets or a 3D convnets model? Because with my actual model I already did the test with that distribution.
You previously mentioned that you ran a 80%/10%/10% trial on a slice basis. Here I am talking about splitting up the data by patient. So, 80 patient volumes for training, 10 patient volumes for validation, and 10 patient volumes for testing. You did mention you had 100 volumes from 100 patients which you had been using for training/validation/testing, correct?
As far as a 2D or 3D model, it would appear you are doing a 2D model since you are talking about slices; in which caseāfor clarityāsplit the previously mentioned volumes into slices. However, you can also try a 3D model and compare the performance to the 2D model. Determining which is appropriate will require experiments with both methods.
1 Like
ok, I agree with the above, I just wanted to know if it was worth trying with 3D convnets given the number of patients, I think at least the comparison will be interesting, Iāll tell you how it was, thank you very much.
1 Like
Hello ,
Iām currently working on a project for segmentation of vertebrae and intervertebral discs using mask rcnn . My dataset has 200 training images and 15 test images of MRI spine. I obtained 2D slices from .nii.gz files of respective images. I manually annotated the training set using Coco-Annotator tool and exported the annotations as a json file in the coco format.
The problem Iām facing now is on reading the json file that contains the annotations. Iām not sure on how to read a json file in the coco format to train the network . Can someone please help me ?
Hi @deepi98,
Can you post some of the json files? and some the data set?
json files can be read with the json library. Which will give you a dictionary object which you can then use.
import json
with open(āXXXX.jsonā, ārā) as myfile:
data=myfile.read()
j_obj = json.loads(data)
Hey @garland3 I solved the issue a while agoā¦but thanks a lotā¦I do have another doubt now. Do you have any idea about how to deploy Mask RCNN trained model as a web application?
Probably deployment would be similar to other app deployments that are described on the forums. What exactly are you having trouble with?