I have just submitted a pull request that addresses this issue. The gist of the problem was that multiple modulemap directories could end up containing the same module.modulemap file, and the swift process got confused presumably because of the duplicated symbols.
I’ve tested a few notebooks and it works for me, but it would be great if someone else could take a look.
One way I found to cause a reproducible bad state kernel was to install SwiftVips in a notebook, and then run the installation cells in 02_fully_connected.
Edit: @marcrasi I forked my branch before your latest merge, sorry about that!