@Dina IIRC people have done this with language models before (and here ULM-FiT). See here:
3 Likes
I would look into Recurrent Neural Networks (in particular, LSTMs) for this problem first. They are kind of an old technique, but they are simpler to use then more modern solutions and they work well on this kind of problems.
Some resources:
Long-Short Term Memory network
https://colah.github.io/posts/2015-08-Understanding-LSTMs/
This is a GREAT presentation to get an intuition, before you dive into the details: https://livefreeordichotomize.com/2017/11/08/lstm-neural-nets-as-told-by-baseball/
For your problem, since you want to predict in the middle of the sequence, you can use a bi-directional LSTM:
2 Likes
Iām assuming that the lack of responses means the project group sessions are done for now. Does everyone meetup in the smaller study groups? Is there a āmeta-threadā with a list of the groups or do I just dig through the forum? Thanks.
3 Likes
You could look at the Source Code Study Group
Also, the Fastbook Study Group
Note that both these study groups generally have advanced discussions.
2 Likes
Would anyone be interested in teaming with me on the flower classification Kaggle competition https://www.kaggle.com/c/flower-classification-with-tpus as a group project? We can compare how fastai2 does with GPUs and TPUs, and we might be able to compare it to TensorFlow. We can also try techniques such as data augmentation, GANs for semi-supervised learning, and label smoothing https://towardsdatascience.com/what-is-label-smoothing-108debd7ef06
1 Like
Looks interesting. Happy to team up. But looks like FastAI still doesnāt work with TPUs yet (?). This competition seems to be geared towards TPU usage. Seems like Pytorch started supporting TPUs recently https://discuss.pytorch.org/t/pytorch-tpu-support/25504
1 Like
Best case scenario, we can test on fastai and then reproduce on TensorFlow. Google would at least appreciate the feedback in knowing where TensorFlow is falling short- for instance, it absorbed Keras to improve usability. Worst case, we can always run the fastai code on GPUs and then export the model to run on TPUs or even CPUs. It seems like Kaggleās flower dataset is a good one to test our knowledge from the first few classes, even if itās not quite what Google was looking for when it started this competition.
1 Like
sounds good! Happy to collaborate ā Have you started working on this already?
Thanks for the info.
1 Like
I have not started but plan to soon. I just formed a team. What is your Kaggle user name, so I can invite you?
Awesome. Youāll need to form a team in order to merge. I went with āFast Teamā- seems simple enough. You can merge with that team, or you can tell me your team name and I can try merging.
Hey David, Iām interested. I also want to have a deeper understanding of their approach
1 Like
Hey Kofi, great!
Iāve been thinking that this might be the best approach in terms of learning:
https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
I think he provides a lot of information, some of which I donāt understand, but all of which seems really interesting. For instance:
āNote that we only go up to the last convolutional layer ā we donāt include fully-connected layers. The reason is that adding the fully connected layers forces you to use a fixed input size for the model (224x224, the original ImageNet format). By only keeping the convolutional modules, our model can be adapted to arbitrary input sizes.ā
This makes me think of many questions, including:
Wait wut? The last convolutional layer is fully connected? Last in what direction? If itās the one before the targetā¦ how is it different from all the others? In the diagrams Iāve seen, each layer is fully connected, which means each node in this layer n gets inputs from every node in layer n-1.
āadding the fully connected layers forces you to use a fixed input size for the modelā I donāt understand this at all. I believe he wrote somewhere else that in Keras, each layer is aware of how many inputs are coming in, so it ādoes it automatically for youā. So that should be able to be turned on or off, I would think. And then what does it mean to not be fully connected?
At any rate, how do you feel about trying to get this understood and working?
Thanks,
David
Iāve just retired from my work on Masks4All so I can focus full-time on fast.ai again, and Iām planning to use Discord as a regular hangout place. So please drop by and hang out if youāre interested in being involved with rolling out fastai2, the course, etc. Iām hoping to do some screen sharing as well, although havenāt set up specific times yet.
Hereās the Discord:
16 Likes
Nice! I will drop by for sure. BTW, I forgot the āgo liveā date of the 2020 courseā¦is it still September or did it change?
this link shows an invalid link in discord