In machine learning, the honest answer to “why did the model do that?” remains “we can’t really tell”. That may be changing, and below is an interesting read. I think it’s intriguing for anyone who works on causality.
In management research, but also in industrial practice, I often see explainability tools treated as if they answered the “why?”. Shapley values, LIME, permutation importance, partial dependence plots, attention maps. All of them point at the inputs that mattered for an output, but none of them tell you what happens if you intervene: on a part inside the model, or on the world the data came from.
While explainability changes the inputs (the variables) and watches the output, interpretability changes a part inside the model (the function itself) and watches the output. The trouble with neural networks is that a concept like “car” has been spread across many neurons, and each neuron served many concepts, so switching one off told you little.
A new essay, covered in the New York Times this week, argues it might be less messy than we thought. The results are preliminary and contested, but in large models, most neurons seem to have one main job. If the claims hold even in part, switching a neuron off then becomes an intervention on something you can name. For causal inference that’s interesting, because normally we observe data and have to guess at the structure behind it (causal discovery).
Now, does theory still matter, then? Likely more than before. Machines can propose hypotheses and test them faster than we can read them. What they can’t tell us is which of the many theories that fit the data is the one we need, because that depends on what we want to test. It’s new territory, again moving the bottleneck from computing answers to choosing the right questions, and nobody has a benchmark for that yet.
First published on LinkedIn.