I would like to build a classification model based on a given data set, which can be of shape like [50000,101,1]. I created a simple architecture, with two levels of convolutional layer, followed by maxpooling, then dropout, then a fully connected layer, something like that.
The problem with this data set, is that the element can be of very small numerical values, such as -3.825970e-09, or -8.64723500e-12, etc.
I am worrying that, just as other numerical problems, this kind of small values can cause the numerical issues. Are there any data pre-processing techniques that we can adopt for this kind of values.
The attached is an example of input data vector.
I never had to deal with such small numbers, but only the exact opposite. I experimented with feature scaling https://en.wikipedia.org/wiki/Feature_scaling as well as with using log1p and expm1. In my case, I was able to significantly improve the training time. You simply have to try out whether it helps you to make better predictions or to lower the training time.
Maybe it helps to show you what I did:
My data was between 0 and 10e10 and the training time was improved a lot by simply using this kind of preprocessing:
input = log1p(input)
A further, but less significant improvement was to scale to make sure the maximum value 10e10 becomes about 1 after the preprocessing. Because log1p(10e10) is approximately 23, my preprocessing now looks like this:
input = log1p(input) / 23
Later on I calculated the mean and variance of the training data, as well as the minimum and maximum to find out whether some kind of simple feature scaling would also work, but this slowed the training time down again.
Experimenting with the mean and variance or the minimum and maximum of the log1p(input) gave me slightly worse than log1p(input) / 23, that’s why I was not using that either.