I don’t think I can speak about what in general would make sense to do (I don’t think myself competent to speak to that) but I can tell you about what I am doing. For me, its pytorch all the way. I tried to take a look at tensorflow, but didn’t find the experience too appealing. From what I remember about keras - have not used it for a very long time - I hated the level of indirection, that my data / parameters lived somewhere but I couldn’t access it directly from python.
I spent quite a bit of time using pytorch and I feel at home in this ecosystem. I can relate to your concerns regarding the fastai library but I think that despite it quickly evolving and maybe there not being there many examples of its usage out there yet, it fits well with what I want to do and how. I have so many plans and ideas involving this specific set of tools.
But this is all very personal. I don’t claim any superpowers and I can relate to many of the struggles you mention. Above all, my choice feels right for me and I have now gathered enough experience to feel very strongly about what I want or don’t want to do.
Whatever choice you feel is right, you should embrace it and head down that path and you should not feel sorry about that. The worse thing one can do is probably trying to do something some other person could tell them to do while they feel like as if they are banging their head against the wall taking this approach and not making progress.
It’s all very individual and situation dependent and I bet you yourself know the answer best! 
That is on the framework bit - for learning, I don’t know any other community that would even come close to how supportive this community is, but I don’t think using any specific set of tools is how one becomes a member.