Human in the loop AI


I’m interested in the topic of “human in the loop AI” approaches, and since I didn’t find a lot of info in the forum I thought I would start a thread. I think humans may have a role to play to fix some of AI shortcomings in production: in terms of raw accuracy improvements, but also maybe more importantly in terms of control for bias and other negative outcomes as introduced in lesson 6.

Do you have papers, articles or books on this subject to recommend? Ultimately, do you have practical suggestions to implement any of these approaches (with / PyTorch)?

Here are my notes so far, I will be updating them:

Human client:
Human provides feedback and corrections on AI’s predictions.
[This topic is vastly covered for recommender systems as it’s their main data source, but not so much for other kinds of AI problems]

Human operator:
Human provides answers when model predictions have low confidence

Human auditor:
Human tests the model against errors, bias, etc: