NOTE: not entirely sure to put this here, in Applications, or Theory, since it feels like it touches all of them. This is part question, part open-discussion.
I’ve been working on repurposing RetinaNet as a pilot-consciousness detector today and there’s a thought I want to throw up here before I go to bed. Can the fast.ai library support run-time data creation? – Would it be a nightmare to implement?
ie: could we create temporary training examples inside the model/learner while it’s running?
What I mean is: I’m thinking of using the bounding-boxes created by RetinaNet (in Keras right now) to crop images and feed those into a classifier. In some cases you have (well, I do) a training-set image with more than one example of the thing you’re detecting (in my case: two views of a person). The bounding-box part of the model will identify two boxes in those cases. Since I want to crop those images and feed them into a classifier, I have an issue if I’m training on file 0001.jpg
and I have two parts of it to crop.
I see 3 immediate options at this point:
- Throw both cropped sub-images through the classifier and take the average of their predictions (is this possible? feels like it’d require modifying the underlying code)
- Take the first or most-confident bounding-box “prediction” and just send that through
- Treat the 2 sub-images as separate training-images with the same label.
Now… there’s another way entirely to go about this, which may be more in line with the way fast.ai is currently handling things, and that’s to run a separate pass with the ‘cropper’ network and save all cropped images to a tmp/
folder (keeping track of labels), and then running that cropped/modified/dynamic dataset through the classifier.
My mind likes the idea of handling the crops on the fly, but I think a more implementable solution is creating the tmp/
folder.
Anyway, there are some thoughts, I’ll probably get to work on one of them tomorrow / this week.
Links:
Keras RetinaNet – Delft Robotics
Focal Loss for Dense Object Detection (RetinaNet paper)
Here’s the notebook I was working on, for visual reference. Be warned, it’s not meant to be pretty, but does show the ‘discovery path’, and examples towards the bottom.