I was using 1.1.12, which I see is before you patched this. So I upgraded to current (1.1.14).
nbdev_build_lib respects the flag: toggling it changes recursion.
nbdev_build_docs ignores it: always recurses.
Example with recursive = False:
notebooks>touch example.ipynb # Create empty file
notebooks>mv example.ipynb exploratory/ # Move it to subdir
example.ipynb -> exploratory/example.ipynb
(peregrine) notebooks>nbdev_build_docs # Run build_docs
...
converting: /Volumes/Peregrine/pg_tools/code/notebooks/exploratory/example.ipynb
Notebook does not appear to be JSON: ''...
...
Conversion failed on the following:
example.ipynb