Bcolz.open() error: No such file or directory file.dat/meta/sizes

Part 1 (2017)
1

Hi,

I’m wondering if anyone can help me. I’m using a jupyter notebook with miniconda on a linux server. I have created a bcolz array using the following:

c = bcolz.carray(x, rootdir=path+'x.dat')

I am then trying to open it (too big to load into RAM) using:

X = bcolz.open(path+'x.dat', mode='r')

This returns the following error:

bcolz/carray_ext.pyx in bcolz.carray_ext.carray.__cinit__ (bcolz/carray_ext.c:14879)()
bcolz/carray_ext.pyx in bcolz.carray_ext.carray.__read_meta (bcolz.carray_ext.c:19583)()

FileNotFoundError: [Errno 2] No such file or directory: [path to the file]/meta/sizes

I have checked the other thread on this issue on the fast.ai forums and googled but cannot find a working answer.

I have doubled checked the path and have checked manually to see if the file does exist. Within the x.dat directory there is indeed a meta sub-directory with a sizes file inside. I also wondered if it could be a read access permission problem and although the file does exist it just can’t read it, but I just successfully wrote the above file to the drive so surely it cannot be that?

I then thought it could be a file corruption problem, but I wrote another small test carray out the same way as above and experiencing the same error when attempting to open that one. This makes me think it is something else.

Any insight would be very helpful.

Thanks

1 Like
2

Which version of bcolz are you using?

I didn’t get any errors running the code below:

Code

import sys
import numpy as np
import bcolz

print("python", sys.version_info[:3])
print("numpy", np.__version__)
print("bcolz", bcolz.__version__)

x = np.random.uniform(size=(2, 2))
c = bcolz.carray(x, rootdir='x.dat', mode='w')
y = bcolz.open('x.dat', mode='r')
y

Output

python (3, 6, 0)
numpy 1.11.3
bcolz 1.1.2

carray((2, 2), float64)
  nbytes := 32; cbytes := 16.00 KB; ratio: 0.00
  cparams := cparams(clevel=5, shuffle=1, cname='lz4', quantize=0)
  chunklen := 1024; chunksize: 16384; blocksize: 0
  rootdir := 'x.dat'
  mode    := 'r'
[[ 0.77617483  0.34371131]
 [ 0.45106069  0.9178579 ]]

I’m on Ubuntu 16.04. I’m also using Anaconda, and ran the code in Jupyter notebook.

Edit:

Try running c.flush() before trying to open x.dat.

3

Try using bcolzArrayIterator() used in lesson 10 in part 2 of the course.

4

Hi,

Thanks for the help.

I forgot to include flush() in my example, but I was trying with that.

I was using google compute cloud and after reading all I could and still not being able to open the array I switched over to AWS. Running the same notebook there, no edits, worked fine. Everything works as expected. I presume there must have been some sort of glitch regarding disk mounting or disk storage or something.

1 Like
5

That’s strange. Never heard that before.

6

Hi there. I am trying lession1.ipynb in Colab, and have encountered a similar error. Haven’t found a solution.

!pip install "fastai==0.7.0"
!apt-get -qq install -y libsm6 libxext6 && pip install -q -U opencv-python
import cv2
from os import path
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())

accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'

!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.3.0.post4-{platform}-linux_x86_64.whl torchvision
!pip install Pillow>=4.1.1
!pip install image
!pip install torchtext==0.2.3

!pip install seaborn --upgrade matplotlib
!pip install seaborn --upgrade pandas
!pip install seaborn --upgrade pillow
!pip install seaborn --upgrade plotly

When running the following code, the error occurred.

arch=resnet34
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(arch, data, precompute=True)
learn.fit(0.01, 2)

The full error message is listed as bellow.

OSError                                   Traceback (most recent call last)
<ipython-input-17-e6c87b20ce86> in <module>()
      1 arch=resnet34
      2 data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
----> 3 learn = ConvLearner.pretrained(arch, data, precompute=True)
      4 learn.fit(0.01, 2)

/usr/local/lib/python3.6/dist-packages/fastai/conv_learner.py in pretrained(cls, f, data, ps, xtra_fc, xtra_cut, **kwargs)
     93     def pretrained(cls, f, data, ps=None, xtra_fc=None, xtra_cut=0, **kwargs):
     94         models = ConvnetBuilder(f, data.c, data.is_multi, data.is_reg, ps=ps, xtra_fc=xtra_fc, xtra_cut=xtra_cut)
---> 95         return cls(data, models, **kwargs)
     96 
     97     @property

/usr/local/lib/python3.6/dist-packages/fastai/conv_learner.py in __init__(self, data, models, precompute, **kwargs)
     86         elif self.metrics is None:
     87             self.metrics = [accuracy_multi] if self.data.is_multi else [accuracy]
---> 88         if precompute: self.save_fc1()
     89         self.freeze()
     90         self.precompute = precompute

/usr/local/lib/python3.6/dist-packages/fastai/conv_learner.py in save_fc1(self)
    128 
    129     def save_fc1(self):
--> 130         self.get_activations()
    131         act, val_act, test_act = self.activations
    132         m=self.models.top_model

/usr/local/lib/python3.6/dist-packages/fastai/conv_learner.py in get_activations(self, force)
    123         names = [os.path.join(self.tmp_path, p+tmpl) for p in ('x_act', 'x_act_val', 'x_act_test')]
    124         if os.path.exists(names[0]) and not force:
--> 125             self.activations = [bcolz.open(p) for p in names]
    126         else:
    127             self.activations = [self.create_empty_bcolz(self.models.nf,n) for n in names]

/usr/local/lib/python3.6/dist-packages/fastai/conv_learner.py in <listcomp>(.0)
    123         names = [os.path.join(self.tmp_path, p+tmpl) for p in ('x_act', 'x_act_val', 'x_act_test')]
    124         if os.path.exists(names[0]) and not force:
--> 125             self.activations = [bcolz.open(p) for p in names]
    126         else:
    127             self.activations = [self.create_empty_bcolz(self.models.nf,n) for n in names]

/usr/local/lib/python3.6/dist-packages/bcolz/toplevel.py in open(rootdir, mode)
    138         return bcolz.ctable(rootdir=rootdir, mode=mode)
    139     else:
--> 140         return bcolz.carray(rootdir=rootdir, mode=mode)
    141 
    142 

bcolz/carray_ext.pyx in bcolz.carray_ext.carray.__cinit__()

bcolz/carray_ext.pyx in bcolz.carray_ext.carray._open_carray()

OSError: data directory does not exist

Would appreciate any help.

For more information: https://github.com/Nov05/fastai/blob/master/colab/lesson1_bcolz_error.ipynb