Overpad image regions which are not interesting

Hello guys, my first post here:
As I have images with a size of 3x512x640, my first step is to pad them with zeros to a square size of 3x640x640. As I am using transfer learning (resnet34) I will feed the network with image sizes of 3x640x640. In my case, it’s not possible to resize the image (too much information would be lost) or crop parts of it to get to common image sizes like e.g. 3x224x224. Considering the fact, that the important region of every image lies always in a horizontal bar area (= red area in the following picture)

I was wondering if there is any procedure to just “overpad” the not reddish area with for example zeros?