Running through my learning again after setting val_idxs = [0] (i.e. just one validation file), I still am getting validation errors and predictions that look like the previous information when I had a full data set - is this expected? Is this validation against my one and only validation file?
Also, should I be looking at the error rate and accuracy data and choosing a place to stop the learning, as I can see that after the second run I am increasingly overfitting and my accuracy is getting worse, so going all the way through 3 runs may not be desirable? Its very hard to know what is really happening without a validation set!
# step 1
learn.fit(1e-2, 7)
[ 0. 0.22371 0.22195 0.94088]
[ 1. 0.20968 0.2259 0.93747]
[ 2. 0.20207 0.22398 0.93844]
[ 3. 0.20532 0.22566 0.93939]
[ 4. 0.18854 0.22653 0.93698]
[ 5. 0.20381 0.22526 0.94088]
[ 6. 0.21357 0.22947 0.93597]
# step 2 -- should I have stopped after this?
learn.fit(1e-2, 3, cycle_len=2, cycle_mult = 2)
[ 0. 0.17768 0.22844 0.93844]
[ 1. 0.17088 0.23041 0.93695]
[ 2. 0.16777 0.23185 0.93796]
[ 6. 0.17352 0.23387 0.93698]
[ 7. 0.16513 0.22885 0.93646]
[ 8. 0.16994 0.23512 0.93792]
[ 9. 0.16108 0.23063 0.93991]
[ 10. 0.15742 0.23026 0.93939]
[ 11. 0.14899 0.22877 0.93991]
[ 12. 0.14532 0.23005 0.94137]
[ 13. 0.16061 0.22951 0.9404 ]
# step 3 - carry on, even though extreme overfitting???
learn.fit(1e-2, 3, cycle_len=1, cycle_mult = 2)
[ 0. 0.16628 0.23203 0.93503]
[ 1. 0.15619 0.23206 0.93646]
[ 2. 0.14303 0.23088 0.93548]
[ 3. 0.15428 0.23497 0.93796]
[ 4. 0.15449 0.23107 0.93841]
[ 5. 0.1584 0.23028 0.93841]
[ 6. 0.14592 0.2302 0.93942]