So I am looking to do something between a using a language model and semantic search. Lets say I give a passage and ask specific questions about the passage- similar to what you would see in SAT/ GRE/ GMAT exams/ elementary reading comprehension. What would be a good starting point to start building something like this myself?
For instance here is a passage from Little red riding hood:
One day her mother said to her, “Come, Little Red Riding Hood, here is a piece of cake and a bottle of wine. Take them to your grandmother, she is ill and weak, and they will do her good. Set out before it gets hot, and when you are going, walk nicely and quietly and do not run off the path, or you may fall and break the bottle, and then your grandmother will get nothing. And when you go into her room, don’t forget to say, good-morning, and don’t peep into every corner before you do it.”
This is what currently ChatGPT does (well); its answer, by just copying and pasting your text, is “Little Red Riding Hood’s grandmother is ill and weak.” . So I would start reading about ChatGPT approach (it’s not the only one, but likely is SOTA now).
Yes thanks for the response. In fact, chatgpt was what got me thinking about this problem and how it can be approached if I were to start with open source language models and want to time to my use cases. While I am in awe of chatgpt for this use case and what is possible, for the scale at which I would ideally want to implement for myself, chatgpt fine tuning becomes too unaffordable. Also , I am only going to do this for a niche area and don’t require the power and might of gpt 3.
How would you recommend I approach this. Any reading material/ pointing in the right direction will help.
PS. I asked the same question on how to approach this to chatgpt and this was the answer I got. Does this sound right @the more advanced fastai patrons
Chat gpt:
One good starting point to build something like this using Fastai would be to use a pre-trained language model and fine-tune it on a dataset of reading comprehension questions and answers. Here are the general steps you can follow:
Collect a dataset of reading comprehension passages and corresponding questions and answers. This dataset should be large enough to train a deep learning model.
Use a pre-trained language model such as BERT or RoBERTa, and fine-tune it on your dataset using the fastai library. This will allow you to train a model that understands the context and meaning of the passage and can answer questions about it.
Use the fine-tuned model to answer questions about new passages. You can use the model’s output to extract the answer from the passage or to generate new text.
Test your model on a held-out test set to evaluate its performance.
Finally, you could also add a semantic search component to your model, which would allow it to find relevant information in the passage using natural language queries. You can use pre-trained models like Elastic search, or other library like Hugging Face’s transformers to perform this task.