Hi Yacine,
You pass the number of epochs you want the model to train, aka the number of times the models gets to improve using the entire dataset.
For more details, have a look at my answer here: "fine_tune" vs. "fit_one_cycle" - #7 by zerotosingularity
thanks a lot ! checking your linked answer right now
Small update: in v2.7.12 (PyTorch 2.0.0) you can use the PILImage again to make a prediction:
img_file = "my_image.jpg"
img = PILImage.create(img_file)
learn.predict(img)
or
learn.predict(img_file)is_bird,_,probs = learn.predict(PILImage.create('bird.jpg'))
print(f"This is a: {is_bird}.")
print(f"Probability it's a bird: {probs[0]:.4f}")
For this last cell in Lesson 1’s code, how do we know which index of probs is referring to bird? I tried a very simple change in the notebook by changing all the ‘bird’ to ‘dog’ but to get the relevant prediction I had to use ‘probs[1]’ instead of ‘probs[0]’. What gives?
To match the probability with the corresponding index, change the first line to
is_bird, index, probs = learn.predict(PILImage.create('bird.jpg'))
and in the last line, use that index as {probs[index]:.4f}`.
It will take care of the issue itself without us bothering what to put!
Here is what happens behind the scenes.
I want to create an instance of ResNet for 3D, 1-channel images (MRIs). All the examples and libraries seem to assume 2D and/or 3-channel images.
How do I an instance of ResNet for 3D, 1-channel images?
Perhaps the Faimed3D extensions (or members in that topic can help.)
I was trying to run this cell in the fourth notebook of the book :
#hide_output
im3_t = tensor(im3)
df = pd.DataFrame(im3_t[4:15,4:22])
df.style.set_properties(**{‘font-size’:‘6pt’}).background_gradient(‘Greys’)
and got this error :
AttributeError Traceback (most recent call last)
/usr/lib/python3/dist-packages/IPython/core/formatters.py in call(self, obj)
343 method = get_real_method(obj, self.print_method)
344 if method is not None:
→ 345 return method()
346 return None
347 else:
~/.local/lib/python3.10/site-packages/pandas/io/formats/style.py in repr_html(self)
381 Hooks into Jupyter notebook rich display system, which calls repr_html by
382 default if an object is returned at the end of a cell.
→ 383 “”"
384 if get_option(“styler.render.repr”) == “html”:
385 return self.to_html()
~/.local/lib/python3.10/site-packages/pandas/io/formats/style.py in to_html(self, buf, table_uuid, table_attributes, sparse_index, sparse_columns, bold_headers, caption, max_rows, max_columns, encoding, doctype_html, exclude_styles, **kwargs)
1306 Whether to sparsify the display of a hierarchical index. Setting to False
1307 will display each explicit level element in a hierarchical key for each
→ 1308 column. Defaults to pandas.options.styler.sparse.columns value.
1309
1310 … versionadded:: 1.4.0
~/.local/lib/python3.10/site-packages/pandas/io/formats/style_render.py in _render_html(self, sparse_index, sparse_columns, max_rows, max_cols, **kwargs)
203 Renders the Styler including all applied styles to HTML.
204 Generates a dict with necessary kwargs passed to jinja2 template.
→ 205 “”"
206 d = self._render(sparse_index, sparse_columns, max_rows, max_cols, " ")
207 d.update(kwargs)
~/.local/lib/python3.10/site-packages/pandas/io/formats/style_render.py in _render(self, sparse_index, sparse_columns, max_rows, max_cols, blank)
160 Also extends the ctx and ctx_index attributes with those of concatenated
161 stylers for use within _translate_latex
→ 162 “”"
163 self._compute()
164 dxs = []
~/.local/lib/python3.10/site-packages/pandas/io/formats/style_render.py in _compute(self)
255 self.ctx_columns.clear()
256 r = self
→ 257 for func, args, kwargs in self._todo:
258 r = func(self)(*args, **kwargs)
259 return r
~/.local/lib/python3.10/site-packages/pandas/io/formats/style.py in _apply(self, func, axis, subset, **kwargs)
1665 “hidden_rows”,
1666 “hidden_columns”,
→ 1667 “ctx”,
1668 “ctx_index”,
1669 “ctx_columns”,
~/.local/lib/python3.10/site-packages/pandas/core/frame.py in apply(self, func, axis, raw, result_type, args, **kwargs)
9421 Apply a function along an axis of the DataFrame.
9422
→ 9423 Objects passed to the function are Series objects whose index is
9424 either the DataFrame’s index (axis=0) or the DataFrame’s columns
9425 (axis=1). By default (result_type=None), the final return type
~/.local/lib/python3.10/site-packages/pandas/core/apply.py in apply(self)
676 # “Union[Series, DataFrame, GroupBy[Any], SeriesGroupBy,
677 # DataFrameGroupBy, BaseWindow, Resampler]”; expected “Union[DataFrame,
→ 678 # Series]”
679 return self.obj.index # type:ignore[arg-type]
680
~/.local/lib/python3.10/site-packages/pandas/core/apply.py in apply_standard(self)
796 “”"
797 we have an empty result; at least 1 axis is 0
→ 798
799 we will try to apply the function to an empty
800 series in order to see if this is a reduction function
~/.local/lib/python3.10/site-packages/pandas/core/apply.py in apply_series_generator(self)
812 from pandas import Series
813
→ 814 if not should_reduce:
815 try:
816 if self.axis == 0:
~/.local/lib/python3.10/site-packages/pandas/core/apply.py in f(x)
131
132 self.result_type = result_type
→ 133
134 # curry if needed
135 if (
~/.local/lib/python3.10/site-packages/pandas/io/formats/style.py in _background_gradient(data, cmap, low, high, text_color_threshold, vmin, vmax, gmap, text_only)
3627 -------
3628 self : Styler
→ 3629
3630 See Also
3631 --------
AttributeError: ‘ColormapRegistry’ object has no attribute ‘get_cmap’
Hey, your code looks correct, and the error log indicates that the libraries pandas and matplotlib have difficulties talking to each other. So this seems like a dependency issue. Maybe you don’t have the current versions of libraries installed?
You can update them with !pip install -Uqq pandas matplotlib (here, U means upgrade and qq means do a quite update, ie don’t give me any logs). Run this command in jupyter and try again. Does it work then?
Afaik, there is no pre-built 3D-resnet in fastai (or anywhere else, really).
So you would have to adapt the resnet architecture to use nn.Conv3d instead of nn.Conv2d and train from scratch.
Hello, I cannot start the course from the beginning. When I reach the part of the code 
from duckduckgo_search import ddg_images
from fastcore.all import *
def search_images(term, max_images=200): return L(ddg_images(term, max_results=max_images)).itemgot(‘image’) and the I call : urls = search_images(“dog images”, max_images=10)
print(urls[0]) I get a 403 (Forbidden message back). It has been some days I am stuck here contary to aking even form chatGPT?
The latest version of DuckDuckGo Search API follows a different format.
Can you try the following code snippet?
It worked for me now.
from duckduckgo_search import DDGS
from fastcore.all import *
def search_images(term, max_images=200):
return L(DDGS().images(term, max_results=max_images)).itemgot("image")
urls = search_images("dog images", max_images=10)
print(urls[0])
Hi, I’m new to fastai, I’ve followed the first lesson of the course and as recommended I was trying to do a small project to test what I’ve learned.
To do something slightly different from what shown in the lesson, I tried to create a model that would identify text files containing source code in Python or Go with their programming language.
The problem I’m encountering is that I run out of memory while trying to fine tune the model.
I’ve tried both in a local environment with CUDA enabled (I have an RTX 2060 with 6GB of memory) and in a Paperspace environment.
I’ve tried reducing batch size since I read in a forum that it could help but it didn’t solve the issue.
Is training a model on text just a bad idea for now or can I make it work?
Here’s the error I encounter:
RuntimeError: CUDA out of memory. Tried to allocate 102.00 MiB (GPU 0; 7.92 GiB total capacity; 7.01 GiB already allocated; 51.69 MiB free; 7.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
This is the full example I’m trying to run:
from fastcore.all import *
from fastai.text.all import *
path = Path('govspython')
# Create a DataBlock
code_data = DataBlock(
blocks=(TextBlock.from_folder(path, is_lm=False), CategoryBlock),
get_items=get_text_files,
get_y=parent_label,
splitter=RandomSplitter(valid_pct=0.2) # Use RandomSplitter if you don't have a 'test' folder
)
# Load the data
dls = code_data.dataloaders(path)
# Check the data
dls.show_batch(max_n=5, bs=16)
learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)
learn.fine_tune(4, 1e-2) # <- this is where it breaks
My dataset is contained in the path shown in code, it’s split into 2 folders, each folder contains some “.txt” files.
Files in the first folder are in Go and files in the other folder are in Python.
Hello.
You can try bs= 8, 4, 2
thanks for the help!
After copying my code here and reading it again I think I saw what my mistake was:
I was setting the batch size on .show_batch() rather than on the dataloader itself.
After changing
dls = code_data.dataloaders(path)
to
dls = code_data.dataloaders(path, **bs=8**)
I managed to train my model locally!
Example of output while trying to predict (these are all Go files - pos is the name of the folder containing go files):
crypto_test.txt is pos with prob tensor([0.0483, 0.9517])
test.txt is pos with prob tensor([0.2439, 0.7561])
main.txt is pos with prob tensor([0.1932, 0.8068])
Since I’m here I’ll try asking another question, in the book, while talking about the choice of a GPU server to use it says:
“We maintain a list of our recommended options on the book’s website, so go there now and follow the instructions to get connected to a GPU deep learning server.”
Is this the list he’s referring to?
The link you are referring is 2019 course. This is latest one. But I could not find any such list😔. You are using paperspace right. You can also have a look at https://lightning.ai/
Hey guys,
I want to create the Datasets object manually (and not from DataBlock). As I understand it, Datasets is just 2 Datasets object, 1 for training, and 1 for validating. I can create 2 Datasets with the training X, and y, and the validation X, and y, but up until now I could not combine them into 1 Datasets object, like the one created using DataBlock.
I have also tried creating a Datasets object with Pytorch’s Dataset class, but with no luck either.
(I am using the docs extensively, but I could not find a solution there.)
Any help, or pointing me to the right direction is appreciated.
Thanks,
Robert
I don’t have practical experience with the situation you have described but I prompted Claude with the Datasets source code and your question and asked for a toy example which I have implemented in this Colab notebook in case you find it helpful.
Thanks, it is really helpful. I have landed on a similar solution, I did not skip the DataBlock, but created the splits manually, and passed it to the IndexSplitter.
That said, I would really have been interested in a solution, if any, as to how to create a Datasets object completely manually. But it is possible, that fastai does not allow for that.
Hey guys,
I am in the process of streamlining my fastai workflow. I have noticed that creating a dataset takes quite a long time, and I was wondering whether it is normal.
I am working with time series data, so my input tensor is 3 dimensional. It has around 12000 rows in it. I initialize a DataBlock, and create a Dataset with the dblock.dsets() method. My independent variables are tensors, my targets can be a list, or a tensor as well. Neither of them shows faster workflow. With this setup, it takes around 3-5 minutes to create the Datasets object.
Thanks,
Robert