From what I can tell it’s a wrapper around fastdownload and in the fastdownload docs it states that you can change ‘base’ to any directory you want. By default {base} points to ~/.fastdownload or something
EDIT: But I see what you’re saying, base can’t be reached from untar_data()
Instead of get, use download to download the URL without extracting it, or extract to extract the URL without downloading it (assuming it’s already been downloaded to the archive directory). All of these methods accept a force parameter which will download/extract the archive even if it’s already present.
You can change any or all of the base, archive, and data paths by passing them to FastDownload:
d = FastDownload(base=’~/.mypath’, archive=‘downloaded’, data=‘extracted’)
Jeremy then showed the Image Classifier Cleaner, and Nick said it pays to visually inspect when using these “open” image searches by @JaviNavarro post.
I’m running into difficulty with ImageClassifierCleaner(learn).
delete() works as expected:
for idx in cleaner.delete(): cleaner.fns[idx].unlink()
change(), however, fails:
for idx,cat in cleaner.change(): shutil.move(str(cleaner.fns[idx]), path/cat)
---------------------------------------------------------------------------
Error Traceback (most recent call last)
/tmp/ipykernel_27738/4259621786.py in <module>
----> 1 for idx,cat in cleaner.change(): shutil.move(str(cleaner.fns[idx]), path/cat)
~/anaconda3/lib/python3.9/shutil.py in move(src, dst, copy_function)
810
811 if os.path.exists(real_dst):
--> 812 raise Error("Destination path '%s' already exists" % real_dst)
813 try:
814 os.rename(src, real_dst)
Error: Destination path 'four_seasons/summer/00000129.jpg' already exists
The problem appears to occurs because search_images_ddg() indexes each category independently, starting with 00000000.jpg. This results in duplicate fnames across categories. Accordingly, attempting to change an image from one category to another results in a collision and the code fails.
I can come up with a work around, but wonder if anyone has solved this particular problem?
Maybe there is a utility I missed in the fastai code base. I will look more closely.
Hi @mike.moloch sorry, I totally appreciate Jeremy must get a lot of @ mentions. Is there a better place to post this? I am taking the course in Brisbane and I don’t yet have an intuition with the library to know what common errors are caused by.
Interestingly I don’t see that IndexError.
My environment is a self provisioned VM on GCP.
Its 8 core, 32gb of ram, T4 with Fresh Ubuntu 22.04 and nvidia 470 drivers, python 3.10 and a virtualenv with a fresh install of fastbook and fastai in the directory. I pulled the example notebook mentioned and ran it and it works fine with this hardware and python environment. At first it looked like it was allocating to the MAX of the GPU but now that I have lowered the image size the memory looks well under the limit when the error occurs as per my screenshot above.
I included a full fresh run from top to bottom of my notebook in the git link I posted above to help debug the issue.
The original notebook has multiple classes and uses a few different functions to detect and set them against a dict of codes.
For mine there appears to be only 1 class and I think its represented by 255, while the void is 0.
Why would 255 be out of bounds, surely thats within 8 bit range?
Do I need to change the mask values so that they are say 0 for void and 1 for the first label or something?
I tried to run it without the Normalize function in the DataBlock (which probably uses the wrong stats but that didn’t change anything).
Here is the other code which relates to the mask and codes.
Any help would be greatly appreciated. I searched the notebooks and I can see its not part of fastbook so I understand im going a little off piste here.
Change download_images to include a new parameter start_idx and have download_images return next_idx so a subsequent call to download_images has a new start_idx
No breaking changes for existing users. Some added complexity for multiple categories, but with the return of next_idx, not too bad.
Create a new function to wrap around:
shutil.move(str(cleaner.fns[idx]), path/cat)
This function would check to see if fname exits and if so, created a unique fname. To the extent fnames are being used for labeling this could get complicated.
Create a utility function reindex_dir that would analyze an existing directory and recreate fnames as specified by the function. Perhaps a signature similar to this:
I never really used github desktop (only used to store my notebooks) nor console but I am assuming the github desktop is easier to use.
Can someone please help me create app.py file (I only used notebooks before) using github console? I followed the video but didn’t work.
@orangelmx Can verify this experience – multi GPU training is full of unexpected, non-linear hassles… like race conditions. Very difficult to trace and debug. It’s just too easy to come unstuck in some nonobvious way and run out of talent especially quickly. This is exacerbated if you have two different GPU models; you will experience bottlenecking from the lower-powered GPU and all sorts of other issues
