@orangelmx Can verify this experience – multi GPU training is full of unexpected, non-linear hassles… like race conditions. Very difficult to trace and debug. It’s just too easy to come unstuck in some nonobvious way and run out of talent especially quickly. This is exacerbated if you have two different GPU models; you will experience bottlenecking from the lower-powered GPU and all sorts of other issues
Oh, is it ‘Nguyen van dui’ you’re talking about ? I think I’ve seen it on social media every now and then. Now that you mentioned, on some of the vids/pics, the background tiles do match.
Thanks for resolving this mystery. Not including links here so as not to go too off-topic
To expand on this answer a little bit: Segmentation in fastai is quite picky about the mask format. The background pixels need to have the value of 0 and all the other classes should use subsequent integer values 1, 2, and so forth. This makes inspecting the mask images in an image viewer quite difficult since all your pixels are almost black.
The clever trick @VishnuSubramanian used helps a lot for binary masks (where pixels are either background – black (0) or foreground – white (255)) where you can get away with dividing the mask by 255 to get only 0 or 1 as outputs.
Btw. it would be great if the SegmentationDataloader could handle random colors (with a vocab-like mechanism) or maybe sanity-check the inputs a bit to show better error messages.
While using the pet classifier, I noticed that if I used the whole picture for inference, it classified it as a “cat” but if I used the edit functionality (pencil in the top right corner of the picture) and cropped it just to the head, it correctly identified it as a “dog”.
I thought it was interesting that the classifier took into account various body configurations and the relationships that may/may not exist. OTOH, this dogs ears look very cat-like so it’s impressive that the classifier still picks it as a dog when shown just the head (cropped from the same picture)
I’ve been looking at the docs and experimenting tonight but not coming up with a solution yet @sambit. Using Google Colab I can’t generally see the downloaded file in the file explorer. Only if I open the terminal can I locate it in the hidden folder .fastai.
My interim hack is to move files using the following terminal cell command. eg: path = untar_data(URLs.IMDB_SAMPLE) !mv {path}/'texts.csv' /content/drive/MyDrive/Notebooks/
The biggest issue I am finding with using fastai and Google Colab is that artifacts like trained models are so easily lost. The config.ini folder locations for data, models, archive would work great for a static environment but as far as I understand right now this isn’t possible with Colab. I hope to be proven wrong!
My understanding is that a new instance is a clean slate and you have to start over unless you download or move your artifacts elsewhere and this can be triggered as soon as you close your laptop lid. I’ve used PyTorch Lightning to train models that have been interrupted and restarted using the most recent epoch backup log file, saving time and energy. I am wondering if fastai can or could do this?
I tend to agree with your observations. I have found colab’s drive situation to be, at the very least, confusing. This is not so much a function of fastai but just the way the colab ecosystem is setup. I have not tried to continue with data saved across session boundaries (mostly because of this).
I think this can be mitigated with extra coding and checkpointing and whatnot. I would think that saving it in my google drive would at least keep the downloaded stuff (config.ini mappings notwithstanding) but I have not made a whole lot of effort in this regard because I find dealing with google drive simply too clunky and unweildy.
I might be probably wrong. I feel since there is no data of thrid category, that third category won’t be recognized by model at all. So probably option (c) is my best guess
I mean, you’re doing all those matrix operations, calculating the loss, getting the gradients, adjusting the weights etc. only for the network to learn to predict 0 for the OTHER class, regardless of what the inputs are.
I’d be very surprised if that class activates at all for any image. I haven’t tried this myself yet, but I’d like to know what happens practically.
My intuition says something like
d) [x, y, z] where x + y ~= 1.0 & z ~= 0.0
I’d guess c too - and with different values than 0.5 if the dog or snake training data had houses in the background (hopefully 0.9, 0.1, 0 - as I like my houses without snakes).
Let’s get it to 5 answers because I’m interested in the answer too
I’m guessing c with a small caveat.
(but more like ~some probability of it predicting OTHER ie; non zero)
My intuition says that since there is no “understanding”, the network may activate on things it has learned for the known categories and mistake the OTHER to be that.
@jeremy I implemented a python function reindex(dest, start_idx=0, ext = None) that takes a directory or directory of directories and uniquely reindexes all of the files across all directories. Optionally, a starting index and extension can be provided. Importantly, when it reindexes, it will save the non-numeric stem of the file name to the extent that was used for labeling. I plan to wrap it in a script so it can be used from either the shell or within Jupyter and post it on github.