You could always go deeper into the data, figure out "local groups" and assign group medians to missing values. There's nothing wrong with this strategy. It's just your way of doing it and you can always use cross-validation to see what works.
The premise that you are skewing the data depends on the kind of model you use and the way your model treats the imputed value.
For example, if you use tree-based models, imputing with -1 is a commonly used strategy and one intuition why it works is "your model treats all these missing values as a separate level by itself". It's like one faulty machine in a large production line not recording this variable and hence they are missing in your dataset and your model is treating all these missing values as originating from one unit.
However, if you use linear equation based models (linear / svms / neural networks), a mean / median / 0 based approach is preferred since the linear optimization is severely affected by the imputation process. Hence, as the mean / median doesn't change the distribution of the feature, you could take this approach.