ImportError: cannot import name 'IterableDataset' from 'torch.utils.data'

fastai fastai dev
1

I am using fastai2 on GCP, working through one of the rsna notebooks, and have run into the following error when importing the fastai2 library:

ImportError: cannot import name 'IterableDataset' from 'torch.utils.data'

I did some upgrading of the python packages, currently using pytorch 1.3, and updated to the most recent version of the fastai_dev repo.

I looked at the file associated with the error and there doesn’t seem to be an IterableDataset to be imported:

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/__init__.py

from .sampler import Sampler, SequentialSampler, RandomSampler, SubsetRandomSampler, WeightedRandomSampler, BatchSampler
from .distributed import DistributedSampler
from .dataset import Dataset, TensorDataset, ConcatDataset, Subset, random_split
from .dataloader import DataLoader

Any ideas on where I’ve gone wrong?

3

Are you sure you have PyTorch 1.3 ? I am using it and IterableDataset is correctly imported in __init__.py. Make sure you are not on a wrong git branch or virtual env which would use version 1.1 somehow.

4

I think I’m messed up on environments. If I do pip show torch, it shows that I pytorch 1.3 is installed, but if I do pip3 show torch, it shows I have version pytorch 1.1 installed. And If I go to where the torch package referenced in the error message I got is intalled, /opt/anaconda3/lib/python3.7/site-packages/torch and look at the version.py file, it shows the version is 1.0.1.

I’m going to look for some resources that describe how to manage environments. I am a bit confused right now about how to determine which version of python is being used and which versions of packages are used between what is native on the server and in either conda or virtual environments.

Also, jupyter is run from systemctl and it isn’t clear to me what environment that service uses. It seems like it would use what is native to the server, but not sure if it picks up the active environment is the service is restarted. Is there a way to specify an environment for the service?

5

Ok, if you edit /lib/systemd/system/jupyter.service and change ExecStart to be as follows

ExecStart=/bin/bash --login -c 'conda activate fastai_dev;/opt/anaconda3/bin/jupyter lab --config=/home/jupyter/.jupyter/jupyter_notebook_config.py'

that will set the environment to the fastai_dev one that you created when you cloned the fastai_dev repo and installed the fastai2 package.

You need to run sudo systemctl daemon-reload and then restart the service with sudo systemctl restart jupyter to get the new file to be active.

Unfortunately, that didn’t solve my problem. If I do conda list torch in the notebook it says that pytorch version 1.3 is installed.

Here is sys.path:

['/home/jupyter/tutorials/fastai',
 '/opt/anaconda3/lib/python37.zip',
 '/opt/anaconda3/lib/python3.7',
 '/opt/anaconda3/lib/python3.7/lib-dynload',
 '',
 '/opt/anaconda3/lib/python3.7/site-packages',
 '/home/jupyter/tutorials/fastai/fastai_dev',
 '/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions',
 '/home/jupyter/.ipython']
6

I ended up stopping the systemd service, activating the fastai_dev environment, installing the required packages and then starting jupyter notebook manually.

7

Did this last solution work ? In theory, the best way to have anaconda envs work correctly is to have no previous python installation. As for notebooks, I suggest you install nb_conda on all your environments, as jupyter doesn’t work well with conda envs (or at least used to, no idea if they patched it). With it, you’ll be able to chose on which env you want to start a kernel. From your sys.path, it doesn’t look like your kernel is using your env.

8

@jeremy @Caleb

I have installed Ubuntu 20.04 LTS. Then installed conda environment. Then installed fastai using fastchan channel. And yet I get this error. cudatoolkit and pytorch-gpu are listed in installed packages in the environment which were all installed using fastchan. Please help. I dont see any way out.

ImportError Traceback (most recent call last)
/tmp/ipykernel_61183/233410589.py in
----> 1 from fastai.vision.all import *

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/fastai/vision/all.py in
----> 1 from . import models
2 from …basics import *
3 from …callback.all import *
4 from .augment import *
5 from .core import *

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/fastai/vision/models/init.py in
----> 1 from . import xresnet
2 from . import unet
3 from .tvm import *

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/fastai/vision/models/xresnet.py in
10
11 # Cell
—> 12 from …torch_basics import *
13 from torchvision.models.utils import load_state_dict_from_url
14

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/fastai/torch_basics.py in
8
9 from .imports import *
—> 10 from .torch_imports import *
11 from .torch_core import *
12 from .layers import *

~/anaconda3/envs/fastai3/lib/python3.7/site-packages/fastai/torch_imports.py in
5 import torch.nn.functional as F
6 from torch.utils.data import SequentialSampler,RandomSampler,Sampler,BatchSampler
----> 7 from torch.utils.data import IterableDataset,get_worker_info
8 from torch.utils.data._utils.collate import default_collate,default_convert
9

ImportError: cannot import name ‘IterableDataset’ from ‘torch.utils.data’ (/home/armankumar/anaconda3/envs/fastai3/lib/python3.7/site-packages/torch/utils/data/init.py)

10

I am getting the same error. Have you solved your issue?

11

Yes. I have solved. It seems that pytorch-cpu version was being used (Despite gpu drivers been proprly used), so I was getting error. I gad created new environment without any package. Then I had run ‘conda install -c fastchan fastai’. And this created problem.

Then I created a new environment. And first installed pytorch using ‘conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia’. Then, ran ‘conda install -c fastchan fastai’. And now, problem solved.

1 Like
12

But make sure your nvidia drivers are installed properly. Take help from Troubleshooting | fastai.

13

Thank You!