With regards to analyzing predictions of conv nets, there are some nice examples of how to go about this in the lesson 1 notebook (where we look at the images that our algorithm struggles with, the confusion matrix, etc). I also remember seeing a nice visual analysis somewhere of model performance done by @sermakarevich for one of the kaggle comps I think
Ah here it is: Kaggle Comp: Plant Seedlings Classification
There is also the really cool vgg cam sort of thing that I believe is covered in the last lecture of p1 v2 but I do not know for sure as I have not gotten to that lecture yet (a similar approach is outlined in the last lecture of p1 v1 and it is quite impressive to be able to do this!).
In general, there are many things one can do to understand the performance of conv nets, however the techniques that Jeremy outlines specific to random forests will not translate 1-1 to analyzing CNNs.
Not sure if this answers your question though - if not, please let me know.