What can be helpful is seeing here the time is going, whether your are GPU bound (your GPU is fully under load, (nearly) all of the time, using all of its memory) or whether you are CPU bound. There are probably more advanced ways of going about this but this is how I do it.
I use tmux, so I open two new panes. In one, I run nvidia-smi -l - this regularly polls the GPU for stats (other people swear by nvidia-smi dmon but I often find nvidia-smi -l easier to read. In the other pane I start htop(this gives me my RAM/swap and CPU utilization). These stats generally suffice for figuring out how I am using my machine, whether I am using it to full potential. Sometimes, when I do something io heavy, I would also useiotop`.
As for colab, even PRO comes with only two vCPUs. I am not sure if this is what is happening here, but for many scenarios this can be a very limiting factor. File access times from google drive have also been slow for me - its best to copy the data locally before training and when moving files around its much faster to move one single file than multiple small ones.
@radek how would you rate nvtop to monitor your GPU compared to nvidia-smi -l ? Its the main thing I monitor for gpu usuage, but haven’t tried nvidia-smi -l or htop
MK-ResNeXt (Mixed Depthwise Convolutional Kernel or Multi-Kernel ResNeXt) modifies X-ResNeXt by replacing some convolutions with mixed depthwise convolutions (MDConv), which were proposed by Mingxing Tan & Quoc V. Le for MixNet.
The notebook contains two early variants, D & E, each with multiple configurations which usually score over 78%.
I also did some unoptimized tests at 20 epochs with MK-ResNeXt and my prior tweaked X-ResNeXt submission. MK-ResNeXt performed better but the lead dropped to 0.5-1%, depending on the configuration.
Based on these current experiments, I believe there is more tweaking to be done to find the best performing configuration. If anyone knows of good resources that cover handcrafting neural network architectures please send them my way.
Novel to the best of my knowledge. If anyone knows of an existing paper or implementation, please let me know. It would save me a bunch of work. ↩︎
I’ve been looking in a somewhat orthogonal direction using network deconvolution, which I’ve described briefly in another forum post. Through some early tests, I’ve found that these deconv (FastDeconv implemented from the official repo by Ye et al. 2020) layers can be substituted into the (x)resnet stem, where they can be drop-in replacements for the Conv2d layers and completely obviate BatchNorm layers!
The results are also quite nice, too. Using an xse_resnext34 model with FastDeconv, I’ve gotten 79.92% ± 0.72%. This can be found near the bottom of this notebook. I also think that this model can be combined with your mixed depth-wise kernels, and might achieve some really nice results
I saw the paper and your post on it the other day and was wondering how well network deconvolutions would work in practice. The answer from this competition seems to be pretty well
At least for ResNet, the deconv networks in their repo appear to replace all the conv layers with deconv layers. Have you tried that substitution with X-ResNeXt? If not, that is probably another avenue to explore.
You’re right, deconvs look like something worth experimenting with in MK-ResNeXt. I will take another look at the paper and code. Already have multiple ideas to test.
Also, isn’t your best result 79.95% +/- 0.23% from the deconv xse-resnet50?
I can’t remember if I tried used their models out of the box, but I swear there was a good reason for not doing so! I think when I tried a small version (xse_resnext18) with all deconv layers, I ended up getting ~65% accuracy. But that was again my own implementation, so it’s worth revisiting. I’ll probably do that next.
You’re right, deconvs look like something worth experimenting with in MK-ResNeXt. I will take another look at the paper and code. Already have multiple ideas to test.
Sounds awesome! I look forward to seeing you crush the high score as usual
Also, isn’t your best result 79.95% +/- 0.23% from the deconv xse-resnet50?
I suppose so, but I decided to not include it cause the accuracy is within the xse_resnext34+deconv score, and I had to use mixed precision in order to train it. (I can’t recall if this was permitted by the rules.)
This version of MK-ResNeXt-50F modifies MK-ResNeXt-50E by adding the deconv stem and moving the squeeze and excite layer back to its original location.
I did have one higher result of 83.22% ± 0.14% (Experiment 5.1 here) by adding a deconv stem to MK-ResNeXt-50E, but was unable to reproduce any similar result so am not officially reporting it. For example, a rerun on a different day, Experiment 5.1.2 in the same notebook, resulted with 82.09% ± 0.26%.
Hey guys, so I have been trying to follow this setup for converting tfrecords to images, but I have hit a snag, this produces an error for me: from tfrecord.tfrecord import
The error:
ModuleNotFoundError: No module named ‘tfrecord.tfrecord’
--------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt. To view examples of installing some common dependencies, click the “Open Examples” button below. ---------------------------------------------------------------------------
I’m using colab, and I cloned the repo and installed TensorFlow.
Thank you for pointing me to it 