Kaggle Questions

Is anyone having problems with the files that the kaggle cli is downloading to AWS (M4.large instance)?
I fixed the file structure in the create_folders.py file and now i’m getting an error that says the downloaded files aren’t zip files

I tried to unzip them manually but I’m getting an error as well where unzip doesn’t work test.zip

The error message suggests that the files need to be unzipped in a particular order? I’m getting the same error when apply unzip to train.zip as well.

Not sure why. Can you try ls -l train.zip and see if you get something equivalent of ~500 MB? It worked for me in the t2.micro instance.
To cut costs, I would recommend doing any non deep learning task on a micro instance. Then we can switch the same instance to a p2 and run DL scripts. That way all the data stored under a micro instance can be reused when using the p2 instance. At least thats what I think will work … correct me if I am wrong.

It worked, I had to go to kaggle page with data with browser and I tried to download link from there and then it asked for approval of competition rules again. And then I ran kg download and then the files worked. The download starts with a connection from ww.kaggle.com, but the valid files seem to come from kaggle2.blob.core.windows.net

2 Likes

To be clear - posting your script was a nice idea; I just wanted to make sure that other participants in the course tried themselves first.

1 Like

I would like to know how to submit to Kaggle.
The sample_submission.csv file has 2 column headers id and label. I am supposing id refers to the image id. Does label refer to the probability of prediction or the category of the image?

Also, how do I generate a csv file from the output of vgg.predict?

I am new to Kaggle. So please excuse me if this is something very basic. I am also trying to find out myself.

@swathi.ssunder Looking at sample_submission.csv is a good way of seeing what the data should look like (if the values are probabilities or categories), but make sure that you are using sample_submission.csv for the right competition. Dogs-vs-cats and Dogs-vs-cats-redux-kernels-edition have different required submission formats, and part of the homework is figuring out how to process and format the output correctly.

You can also get info on submissions from looking at the Kaggle Evaluation page for that competition:
Dogs vs Cats Redux: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation
Dogs vs Cats: https://www.kaggle.com/c/dogs-vs-cats/details/evaluation
(again, note that they are different)

As for generating a csv (from a numpy array), can you tell me what you’ve tried searching for so far, or what methods you’ve already tried? I can give you a hint if you are stuck.

I have a Kaggle set up question. I tried pip install kaggle-cli in Cygwin at Windows 7, but I am not sure it was successful. I tried pip install python-lxml, but I am not sure it worked either. What shall I do?

kicho@PC08EYW5 ~
$ pip install kaggle-cli
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting kaggle-cli
Requirement already satisfied (use --upgrade to upgrade): cliff in /usr/lib/python2.7/site-packages (from kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): MechanicalSoup in /usr/lib/python2.7/site-packages (from kaggle-cli)
Collecting lxml (from kaggle-cli)
  Downloading lxml-3.6.4.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 195kB/s
Collecting cssselect (from kaggle-cli)
  Downloading cssselect-1.0.0-py2.py3-none-any.whl
Collecting configparser (from kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): cmd2>=0.6.7 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): PrettyTable<0.8,>=0.7 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): unicodecsv>=0.8.0; python_version < "3.0" in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): pbr>=1.6 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): stevedore>=1.16.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.1.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in /usr/lib/python2.7/site-packages (from MechanicalSoup->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.0 in /usr/lib/python2.7/site-packages (from MechanicalSoup->kaggle-cli)
Installing collected packages: lxml, cssselect, configparser, kaggle-cli
  Running setup.py install for lxml ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xCaOTi/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QWptuE-record/install-record.txt --single-version-externally-managed --compile:
    Building lxml version 3.6.4.
    Building without Cython.
    ERROR: /bin/sh: xslt-config: command not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    running install
    running build
    running build_py
    creating build
    creating build/lib.cygwin-2.6.0-x86_64-2.7
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/cssselect.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/doctestcompare.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/ElementInclude.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/pyclasslookup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/sax.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/_elementpath.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/clean.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/defs.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/diff.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/ElementSoup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/formfill.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/html5parser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/soupparser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_diffcommand.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_html5builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_setmixin.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/lxml.etree.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/includes/c14n.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etreepublic.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/htmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.cygwin-2.6.0-x86_64-2.7
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml
    gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/build=/usr/src/debug/python-2.7.12-1 -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/src/Python-2.7.12=/usr/src/debug/python-2.7.12-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml/lxml.etree.o -w
    unable to execute 'gcc': No such file or directory
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitAWWi_U.c -o tmp/xmlXPathInitAWWi_U.o
    unable to execute 'cc': No such file or directory
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xCaOTi/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QWptuE-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-xCaOTi/lxml/
kicho@PC08EYW5 ~
$ pip install python-lxml
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting python-lxml
  Could not find a version that satisfies the requirement python-lxml (from versions: )
No matching distribution found for python-lxml

@analytics20132333 it would be ‘pip install lxml’, not ‘pip install python-lxml’. Having said that, there’s really no reason to install kaggle cli on Windows - you can just use your browser to submit to Kaggle on Windows.

But I’m guessing this isn’t actually what you want - assuming that you’re doing your GPU training on Linux (i.e. an AWS instance), then you need to be installing kaggle CLI onto your GPU server, not into Cygwin. The purpose of Kaggle CLI is that it’s hard to navigate the Kaggle website and make a submission via your Linux server, so it’s generally easier to to submit through a command line interface (CLI). So you would type ‘pip install kaggle-cli’ into your bash shell on your server.

Does that make sense?

1 Like

When working with the test set:

The get_batches function presumes the passed-in path will contain subdirectories, so you’ll want to create a subdirectory within your test folder (you can name it anything you want), and move all of the images into that subdirectory.

If you’re wondering how to get the filenames for the batched images, there is a .filenames method you can call…

3 Likes

Hi @jeremy! I think it makes sense. I was simply following the assignment instruction. I was at Cygwin level, but I tried again on my AWS instance after reading your comment. I got the same error after trying pip install lxml. Is there a way to make the command line interface installation possible on an AWS instance? Or should I use a browser in Windows?

kicho@PC08EYW5 ~
$ pip install lxml
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
  Downloading lxml-3.6.4.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 276kB/s
Installing collected packages: lxml
  Running setup.py install for lxml ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AeWcP9/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4tvJdQ-record/install-record.txt --single-version-externally-managed --compile:
    Building lxml version 3.6.4.
    Building without Cython.
    ERROR: /bin/sh: xslt-config: command not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    running install
    running build
    running build_py
    creating build
    creating build/lib.cygwin-2.6.0-x86_64-2.7
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/cssselect.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/doctestcompare.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/ElementInclude.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/pyclasslookup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/sax.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/_elementpath.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/clean.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/defs.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/diff.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/ElementSoup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/formfill.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/html5parser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/soupparser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_diffcommand.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_html5builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_setmixin.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/lxml.etree.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/includes/c14n.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etreepublic.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/htmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.cygwin-2.6.0-x86_64-2.7
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml
    gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/build=/usr/src/debug/python-2.7.12-1 -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/src/Python-2.7.12=/usr/src/debug/python-2.7.12-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml/lxml.etree.o -w
    unable to execute 'gcc': No such file or directory
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitd1Hw8A.c -o tmp/xmlXPathInitd1Hw8A.o
    unable to execute 'cc': No such file or directory
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AeWcP9/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4tvJdQ-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-AeWcP9/lxml/

As it mentions in the error ‘make sure the development packages of libxml2 and libxslt are installed’. They should already be part of anaconda - are you using our script to setup your AWS instance? It looks like you are either not using it, or are not connected to your instance, since your prompt above shows ‘kicho@PC08EYW5 ~’, but on our AWS setup your username would be ‘ubuntu’, not ‘kicho’.

On our AWS setup, you should not need to install any prerequisites in order to install kaggle-cli - just ‘pip install kaggle-cli’ should be it.

I was getting an error
ubuntu@ip-10-0-0-5:~/nbs$ kg download
Starting new HTTPS connection (1): www.kaggle.com
’NoneType’ object has no attribute ‘find_all’

There was a typo in the competition name. I was using dogs-vs-cats-redux-kernel-edition instead of dogs-vs-cats-redux-kernels-edition. That fixed it.

Thanks @jeremy for pointing out!

1 Like

Thank you, Jeremy! It works now. I accessed an AWS instance using our script and then pip install kaggle-cli worked. :slight_smile:

1 Like

http://forums.fast.ai/t/kaggle-questions/135/29?u=swathi.ssunder

@rachel - Thank you for pointing me to the evaluation page, where there is info about the submission s.
My question was about how to actually run the trained model on the test data? Was I right in trying to execute vgg.predict on the test data?
When I actually did this, the result was a tuple, which I converted to a numpy array through numpy.asarray and then wrote to a csv file.
I think there is something wrong in this.

Remember that the get_batches method shuffle has a default setting of True.

2 Likes

@swathi.ssunder you’re on the right track, but you’ll need to spend some time looking at the data structure that’s returned from predict(), and figuring out how to modify it to be the correct structure to submit to kaggle. You’ll need to write some code to make these changes.

Give it your best shot, and if you get stuck, let us know exactly what you’ve tried, what you’ve found so far, and what it is that you need to do next and don’t know how to do.

1 Like

http://forums.fast.ai/t/kaggle-questions/135/39?u=swathi.ssunder

@jeremy - thanks for clarifying. sure, I shall do that.

I used SciKit Learn’s StratifiedKFold and generating symlinks in python to dynamically generate the training and validation sets (and reset them, if necessary). I hope that’s ok.

thanks for the tip! also, are the file names the id names?

@melissa.fabros yes that’s exactly right!