Importance of scaling response for regression with vision learner

I am interested in using vision learners for regression tasks. My response is a continuous variable that is natively scaled from 0 to 100, the percentage cover of plants occupying an area. In my experiments I have noticed that scaling of these data is of great importance. For example I have tried taking the log of the response, the square root, the native 0 to 100 integers, and rescaling from 0 to 1. All perform quite poorly, but 0 to 1 is the least bad. I am used to working with shallow tree-based algorithms, such as random forest that are seemingly agnostic to scale of y. Does anyone have insight as to the best scale for response in regression tasks within the fastai framework?