New user, want to train ai to find math

fastai
1

Hello, I’m a new fastai user. I’m going through the Deep Learning for Coders with fastai & PyTorch book right now but ultimately I’d like to create an ai model which is able to recognize math equations found in an image. ie I take a photo of a math textbook, and the AI can find all the math on the page. I’d like the model to be able to generate MathML ( MathML - Wikipedia) for all the identified parts on the page.

Any help would be appreciated in guiding me on how to use fastai to achieve my goal. I was thinking maybe my problem would be an OCR type of problem Building Custom Deep Learning Based OCR models (nanonets.com) but current off the shelf OCR’s are terrible at identifying math. ie x^2 can’t even be accurately identified, not to mention fractions, square roots, sigmas etc

I was thinking that my problem might be able to be solved by using Segmentation ( Computer vision | fastai) If I can train an ai model to recognize each math character then I would need to find the bounding box (not sure how to do that yet) of each character, and then I would have to stitch together a math equation based on the boxes that I have?

Any suggestions or approaches that I could look at?

1 Like
2

Hi Marek

There is a very old paper on ResearchGate Mathematical Formula Recognition using Machine Learning Techniques (2010) which might be a start. I think the issue is not recognizing individuals symbols in isolation but groups of symbols so more like natural language processing of sequences. Deep Learning generally does not understand (the semantics) only the syntax. So it should be able to copy from image to Latex.
Regards Conwyn

3

Do you have a link to that paper on ResearchGate. I can’t seem to find it.

Oh, I just found the paper using google scholar instead. Thanks!