I’m working on a project where where rank notifications in our software. One idea I’ve had is that we should just start with a heuristic as we have some intuition on how to prioritize the notifications. We can then let our users give a thumbs up/thumbs down to give us feedback on whether we were right or not.
Using this feedback data we could then iterate on a machine learning based model and include other feature data we have.
Does the idea to start out with a heuristic make sense? If so how should i measure the performance of the heuristic vs the model. Should I compare recall and performance of the two? I was thinking that I could start out by doing the heuristic and then attempt to build a model that performs better than it.
Does my strategy make sense?