Seeking feedback: Reflection/Ethical Thinking widget

[nbharatula (Nalini)]

Folks, I want to seek early feedback for an idea, please help!

I’ve been thinking about tools that can help trigger some sort of reflection or “ethical thinking” during the model building process. My target “users” are individual data scientists and researchers or AI/ML/DL practitioners. My goal is to make reflection a habit. Seeking feedback on one such (very simple) tool…

Proposal

A set of 5 questions provided as a fastai widget OR a form (html/google). Running the widget triggers the questions, when you answer/submit they become part of your Jupyter notebooks (yay literate programming!) available for all to read.

Now, for the feedback I’m seeking, easy stuff first:

1) Do you think this is valuable?

• Yes
• No

2) As a user, will you actually fill it (once per project)?

• Sure
• Never
• Maybe, if you add/change <pls leave a comment!>

Harder part of the feedback:

Do the 5 questions below (3 in bold are key imo) make sense to you? What do you think I’m asking for? Any key question missing from this? Any other feedback?
(Please leave responses as replies to this post. Thank you!!)

The 5 Reflections/Questions:

1. State ONE target application for this work.
2. Is your data/reward EXACTLY what you want to predict/classify/learn
3. State ONE application of this work that you think is harmful.
4. State who or what is NOT represented in your data but is impacted by the results.
5. These results are only valid FOR <list groups/conditions>.

Please Note

I am intentionally not explaining the questions - I want to see how folks understand them and if that doesn’t meet my intended reflection I will revisit the wording/question.

I am specifically not targeting for this to be useful to big teams/corporations. They need something more intense/comprehensive. The “Model Cards for Models” and “Datasheets for Datasets” papers provide such intense/comprehensive tools. If anyone here works at a company using something like this, please let me know, I’d like to chat more!

I am also not intending to reduce ethics to a checklist or process, though that is a risk. We have to start somewhere, and I think a simple reflection as an individual habit is a good starting point. Not enough, but I think valuable, if you disagree, I want to hear more!

Also, plenty of ideas on what more can be done or how to do it differently - for example we can reduce it to 3 reflections if that increases adoption without loss of meaning, or use an entirely different method to trigger it, can also evolve to be enforceable etc etc. But this is just a POC, so keeping it simple for now.

Proof of Concept

Jupyter Notebook: https://github.com/nalinicommits/reflectionwidget/blob/nbonfastai/nbs/dl1/InitiateReflectionWidget.ipynb
If that doesn’t render, try the gist: https://gist.github.com/nalinicommits/79770dc78e13aa52c22d23c71dd00cfc
If neither renders, try loading the notebook on nbviewer.

Thank you all for your time!

[jamesd (James Dellinger)]

I think it’s great how you’re taking the first step and getting the ball rolling. I agree that it’ll be wonderful to live in a world where consideration of ethics edge-cases and blindspots is also widely found in code comments/notebook cells.

I think I get the gist of question 2, but I’m little fuzzy on its precise meaning. My assumption is that the intention is to spur the practitioner to think along the lines of: “How is your evaluation metric, loss or reward function connected to the problem your product is solving?” Or alternatively, as this good paper put it, “what action they want someone to take as a result of a prediction.”

For question 5, I wonder if there may be some cases where it could feel slightly daunting to provide a comprehensive answer the question as it’s currently phrased. If the goal is to help practitioners cultivate a habit of reflection, I wonder if asking the reverse question might also be just as enlightening while being lighter weight. e.g. “Please list one group/condition where these results would likely be invalid.”

[JennyCai]

Great first step, Nalini! The questions make even more sense if a researcher is doing a project from scratch, from collecting data, thinking about questions that can be answered from the data, up to suggesting applications. I especially appreciate the words “ONE target application” and “ONE application of this work”. They help researchers focus.

It’ll be good that people start to think about these questions early on, say, when they do a Kaggle project. I wonder whether these will help:

1. Provide example answers as a reference. Say pick a Kaggle project to answer these questions.
2. Three is a magical number. How about just the three bolded questions? So people don’t feel overwhelmed. When they start thinking, they are likely to write more.

Your questions will make a model not only more ethical but also more practical. Thanks again for taking the initiatives.

[nbharatula (Nalini)]

@jamesd @JennyCai thank you for the encouragement and for sharing your thoughts!

James, thanks for pointing me to the paper, it’s a great way for product managers to think about in the AI/ML space. My intent with Q2 was slightly different, but perhaps defining an actionable objective may encompass it (not sure). Your rewording also reads larger in scope than what I had intended. My goal with Q2 was to help discover invalid/incomplete associations between input and output. For example, for supervised learning, using features/datasets that are indicative of certain traits and then assuming we’ve learnt to predict those traits is a huge problem. For example, 1) is success in a coding test a sign of a strong candidate? 2) is a movie review really a sign of what movies I like? 3) is a smile really a sign of happiness? 4) is gait a sign of harmful intent/impending action etc. In the case of reward functions, your statement or defining actionable objectives seem more appropriate, though we’ll still have the risk of unintended consequences and gaming of reward functions. Maybe this is all too much to pack into 1 Q&A.

I like your rephrasing for Q5. Originally I thought defining the conditions under which that accuracy was achieved should be simple enough, but I suppose researchers want to believe their models are generalisable beyond those conditions! So forcing a single example where it does not generalise might be a good start!

@JennyCai , thanks for appreciating the choice of “one” to help focus! I agree this exercise is more suited for some projects than others, but at some level it is not about suitability, and more about building muscle memory. I will think more about adding reference examples - has its pros and cons, as in it might narrow the reflection but might also help elicit more participation.

I am also tending towards 3 questions, but I want to let others weigh in as well. So far, judging by views vs votes doesn’t seem like folks are interested in this tool!

Thanks again for engaging with this. I’m open to critical feedback as well, I’d like to think I can handle it!

[rachel (Rachel Thomas)]

Hi @nbharatula,
I like this idea! I would echo some of the suggestions of others. Here are some potential changes in wording (these are just suggestions so feel free to disregard), mostly along the lines of many people may have not considered misuses, proxies, or stakeholders, and softening the wording a bit:

2. Most metrics/rewards are just proxies of what we really care about. What is the true value (usually something that can’t be measured) that you are interested in?
3. How could a bad actor try to weaponize this? Are there any potential applications that you didn’t intend but that could be harmful?
4. List all groups of people who might be impacted this work.

Let us know where you end up taking this idea, because I think it could be helpful!

[nbharatula (Nalini)]

Thank you for the feedback, @rachel. Glad you like the core idea!

I will do a second pass, incorporating all feedback and developing the idea further, and come back with updates… in time.