# Lesson 2 official topic

I might be probably wrong. I feel since there is no data of thrid category, that third category won’t be recognized by model at all. So probably option (c) is my best guess

1 Like

I mean, you’re doing all those matrix operations, calculating the loss, getting the gradients, adjusting the weights etc. only for the network to learn to predict 0 for the OTHER class, regardless of what the inputs are.

I’d be very surprised if that class activates at all for any image. I haven’t tried this myself yet, but I’d like to know what happens practically.

My intuition says something like
d) [x, y, z] where x + y ~= 1.0 & z ~= 0.0

2 Likes

I’d guess c too - and with different values than 0.5 if the dog or snake training data had houses in the background (hopefully 0.9, 0.1, 0 - as I like my houses without snakes).

2 Likes

Let’s get it to 5 answers because I’m interested in the answer too

I’m guessing c with a small caveat.
(but more like ~some probability of it predicting OTHER ie; non zero)

My intuition says that since there is no “understanding”, the network may activate on things it has learned for the known categories and mistake the OTHER to be that.

1 Like

@jeremy I implemented a python function `reindex(dest, start_idx=0, ext = None)` that takes a directory or directory of directories and uniquely reindexes all of the files across all directories. Optionally, a starting index and extension can be provided. Importantly, when it reindexes, it will save the non-numeric stem of the file name to the extent that was used for labeling. I plan to wrap it in a script so it can be used from either the shell or within Jupyter and post it on github.

``````image_dir contains directories of images: dir1/{1.jpg, 2.jpg, 3.jpg}, dir2/{1.jpg, 2.jpg, 3.jpg}, dir3/{1.jpg, 2.jpg, 3.jpg}
usage: reindex(image_dir) -> dir1/{1.jpg, 2.jpg, 3.jpg}, dir2/{4.jpg, 5.jpg, 6.jpg}, dir3/{7.jpg, 8.jpg, 9.jpg}
``````
``````def reindex_dir(dest, start_idx=0, ext = None):
dest = Path(dest)
rnd  = "zsk#@m"
fns,idx  = [], None
for idx, fn in enumerate(filter(Path.is_file, dest.iterdir()), start=start_idx):
suf = ext if ext else fn.suffix.strip('.')
stm = re.compile('[0-9]*\$').sub("",fn.stem.strip('.')) # remove numeric idx from end of stem
fn_new = dest/(f'{stm}{idx}.{suf}')
fn_tmp = dest/(str(idx) + rnd)                         # Need unique fn to  avoid collisions
fns.append([fn_tmp,fn_new])
fn.rename(fn_tmp)
for fn_tmp, fn_new in fns:
fn_tmp.rename(fn_new)
return idx + 1 if idx else 0

def reindex(dest, start_idx=0, ext = None):
dest = Path(dest)
if not(dest.is_dir()): return f'{dest} is not a directory'
idx = start_idx
for d in filter(Path.is_dir, dest.iterdir()):
idx = reindex_dir(d, idx, ext=ext)
``````

2 posts were merged into an existing topic: Help: Using Colab or Kaggle

That should be fixed now - I modified the function to save images with random filenames.

6 Likes

Re : survey Qu1 my answer middle option

If you have an issue in that you feel you are falling behind in your schedule then if you need to speed up that’s fine with me. What I note with V5 is that the lessons are much shorter in duration compared with previous version so I guess there is more pressure to get through all the material, perhaps Iam mistaken.

So fastai lib will decide whether to use the GPU or not during training based on model size and/or dataset number of samples? I was going through the notebook of the 2nd chapter of the book and when training the bear classifier I noticed that `nvidia-smi` showed no activity on the GPU, even though I have a GPU available:

``````=== Software ===
python        : 3.9.12
fastai        : 2.6.0
fastcore      : 1.4.2
fastprogress  : 1.0.2
torch         : 1.11.0+cu102
nvidia driver : 440.64
torch cuda    : 10.2 / is available
torch cudnn   : 7605 / is enabled

=== Hardware ===
nvidia gpus   : 1
torch devices : 1
- gpu0      : Tesla V100-PCIE-16GB

=== Environment ===
platform      : Linux-5.4.86-1.el7.x86_64-x86_64-with-glibc2.17
distro        : #1 SMP Wed Jan 6 09:06:12 UTC 2021
conda env     : Unknown
python        : ../fastai/env/bin/python3
sys.path      : ../fastai/fastbook
../.pyenv/versions/3.9.12/lib/python39.zip
../.pyenv/versions/3.9.12/lib/python3.9

../fastai/env/lib/python3.9/site-packages
``````

15 posts were merged into an existing topic: Help: Creating a dataset, and using Gradio / Spaces

Not sure why, probably have to look into your code/setup because the learner loads the model in GPU, whenever available.

3 Likes

So I flushed my setup and used fastsetup this time, still `nvidia-smi` shows no process running on the GPU. Interestingly, `nvtop` does show memory usage while training the bear classifier.

so, at least, I have the confirmation that the GPU is being used . Thank you for the pointer @imrandude

I am doubting if `torch` is able to recognise the `GPU`. You can easily check it by using this code.

``````import torch
torch.cuda.is_available()
``````

It should return True if it is able to recognize. If it is False, then maybe something went wrong with the installation.

2 Likes

it does return `True`.

I hope you tried `watch nvidia-smi` which refreshes. I am just wondering if you have checked it while the model was not running.

If you have used `watch nvidia-smi`, then I am not sure what is happening

yes… i tried `watch nvidia-smi` and no processes show up. As well, while looking at the help of `nvidia-smi` i saw you can probe in a loop using `-l` or `--loop` and tried that but same result, no processes.

1 Like

Maybe it’s happening too fast? I’d try it with the imdb example in the first notebook with a large batch size (64) and run `nvidia-smi -l` to see if it hits the gpu or gpu ram? on a v100 it should take about 60-120 seconds per epoch IIRC … if it takes much much longer then it’s probably not hitting the GPU

That is a great addition!

I came up with a different solution, which was to create a new function python function reindex, which can be used to create unique pathnames across a set of directories.

`reindex` fixes this problem by uniquely reindexing all files within first-level directories of dest

``````dest contains directories of images: dir1/{1.jpg, 2.jpg, 3.jpg}, dir2/{1.jpg, 2.jpg, 3.jpg}, dir3/{1.jpg, 2.jpg, 3.jpg}
reindex(dest, start_idx=1) -> dir1/{1.jpg, 2.jpg, 3.jpg}, dir2/{4.jpg, 5.jpg, 6.jpg}, dir3/{7.jpg, 8.jpg, 9.jpg}
``````

It can be used from within python or from the shell. Here is how one would use it from the shell:

``````usage: reindex.py [-h] [--start_idx START_IDX] [--ext EXT] dest

Uniquely reindexes all files within first-level directories of `dest`

positional arguments:
dest                  Contains one or more directories

optional arguments:
-h, --help            show this help message and exit
--start_idx START_IDX
Starting index across all files
--ext EXT             Optional file extention

``````

It is on github at fastai_utilities

1 Like

If you are running it inside a container then the process will not be displayed. If you have access to the host you can see the process though.

Ok, that’s a good point, maybe an epoch lasts less than 60 secs for the bear classifier, I’ll try your suggestion. Now, I am sure it is using the GPU as `nvtop` shows memory being used in the GPU the training, as I pointed out previously