Kaggle Questions


(Swathi Shyam Sunder) #28

I would like to know how to submit to Kaggle.
The sample_submission.csv file has 2 column headers id and label. I am supposing id refers to the image id. Does label refer to the probability of prediction or the category of the image?

Also, how do I generate a csv file from the output of vgg.predict?

I am new to Kaggle. So please excuse me if this is something very basic. I am also trying to find out myself.


(Rachel Thomas) #29

@swathi.ssunder Looking at sample_submission.csv is a good way of seeing what the data should look like (if the values are probabilities or categories), but make sure that you are using sample_submission.csv for the right competition. Dogs-vs-cats and Dogs-vs-cats-redux-kernels-edition have different required submission formats, and part of the homework is figuring out how to process and format the output correctly.

You can also get info on submissions from looking at the Kaggle Evaluation page for that competition:
Dogs vs Cats Redux: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation
Dogs vs Cats: https://www.kaggle.com/c/dogs-vs-cats/details/evaluation
(again, note that they are different)

As for generating a csv (from a numpy array), can you tell me what you’ve tried searching for so far, or what methods you’ve already tried? I can give you a hint if you are stuck.


(Kicho) #30

I have a Kaggle set up question. I tried pip install kaggle-cli in Cygwin at Windows 7, but I am not sure it was successful. I tried pip install python-lxml, but I am not sure it worked either. What shall I do?

kicho@PC08EYW5 ~
$ pip install kaggle-cli
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting kaggle-cli
Requirement already satisfied (use --upgrade to upgrade): cliff in /usr/lib/python2.7/site-packages (from kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): MechanicalSoup in /usr/lib/python2.7/site-packages (from kaggle-cli)
Collecting lxml (from kaggle-cli)
  Downloading lxml-3.6.4.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 195kB/s
Collecting cssselect (from kaggle-cli)
  Downloading cssselect-1.0.0-py2.py3-none-any.whl
Collecting configparser (from kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): cmd2>=0.6.7 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): PrettyTable<0.8,>=0.7 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): unicodecsv>=0.8.0; python_version < "3.0" in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): pbr>=1.6 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): stevedore>=1.16.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.1.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in /usr/lib/python2.7/site-packages (from MechanicalSoup->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.0 in /usr/lib/python2.7/site-packages (from MechanicalSoup->kaggle-cli)
Installing collected packages: lxml, cssselect, configparser, kaggle-cli
  Running setup.py install for lxml ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xCaOTi/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QWptuE-record/install-record.txt --single-version-externally-managed --compile:
    Building lxml version 3.6.4.
    Building without Cython.
    ERROR: /bin/sh: xslt-config: command not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    running install
    running build
    running build_py
    creating build
    creating build/lib.cygwin-2.6.0-x86_64-2.7
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/cssselect.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/doctestcompare.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/ElementInclude.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/pyclasslookup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/sax.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/_elementpath.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/clean.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/defs.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/diff.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/ElementSoup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/formfill.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/html5parser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/soupparser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_diffcommand.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_html5builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_setmixin.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/lxml.etree.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/includes/c14n.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etreepublic.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/htmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.cygwin-2.6.0-x86_64-2.7
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml
    gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/build=/usr/src/debug/python-2.7.12-1 -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/src/Python-2.7.12=/usr/src/debug/python-2.7.12-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml/lxml.etree.o -w
    unable to execute 'gcc': No such file or directory
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitAWWi_U.c -o tmp/xmlXPathInitAWWi_U.o
    unable to execute 'cc': No such file or directory
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xCaOTi/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QWptuE-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-xCaOTi/lxml/
kicho@PC08EYW5 ~
$ pip install python-lxml
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting python-lxml
  Could not find a version that satisfies the requirement python-lxml (from versions: )
No matching distribution found for python-lxml


(Jeremy Howard) #31

@analytics20132333 it would be ‘pip install lxml’, not ‘pip install python-lxml’. Having said that, there’s really no reason to install kaggle cli on Windows - you can just use your browser to submit to Kaggle on Windows.

But I’m guessing this isn’t actually what you want - assuming that you’re doing your GPU training on Linux (i.e. an AWS instance), then you need to be installing kaggle CLI onto your GPU server, not into Cygwin. The purpose of Kaggle CLI is that it’s hard to navigate the Kaggle website and make a submission via your Linux server, so it’s generally easier to to submit through a command line interface (CLI). So you would type ‘pip install kaggle-cli’ into your bash shell on your server.

Does that make sense?


(wendydherin) #32

When working with the test set:

The get_batches function presumes the passed-in path will contain subdirectories, so you’ll want to create a subdirectory within your test folder (you can name it anything you want), and move all of the images into that subdirectory.

If you’re wondering how to get the filenames for the batched images, there is a .filenames method you can call…


(Kicho) #33

Hi @jeremy! I think it makes sense. I was simply following the assignment instruction. I was at Cygwin level, but I tried again on my AWS instance after reading your comment. I got the same error after trying pip install lxml. Is there a way to make the command line interface installation possible on an AWS instance? Or should I use a browser in Windows?

kicho@PC08EYW5 ~
$ pip install lxml
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
  Downloading lxml-3.6.4.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 276kB/s
Installing collected packages: lxml
  Running setup.py install for lxml ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AeWcP9/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4tvJdQ-record/install-record.txt --single-version-externally-managed --compile:
    Building lxml version 3.6.4.
    Building without Cython.
    ERROR: /bin/sh: xslt-config: command not found

    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    running install
    running build
    running build_py
    creating build
    creating build/lib.cygwin-2.6.0-x86_64-2.7
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/cssselect.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/doctestcompare.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/ElementInclude.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/pyclasslookup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/sax.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/_elementpath.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/clean.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/defs.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/diff.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/ElementSoup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/formfill.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/html5parser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/soupparser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_diffcommand.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_html5builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/_setmixin.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    copying src/lxml/html/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
    copying src/lxml/lxml.etree.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
    copying src/lxml/includes/c14n.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etreepublic.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/htmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
    creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.cygwin-2.6.0-x86_64-2.7
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src
    creating build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml
    gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/build=/usr/src/debug/python-2.7.12-1 -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/src/Python-2.7.12=/usr/src/debug/python-2.7.12-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml/lxml.etree.o -w
    unable to execute 'gcc': No such file or directory
    Compile failed: command 'gcc' failed with exit status 1
    creating tmp
    cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitd1Hw8A.c -o tmp/xmlXPathInitd1Hw8A.o
    unable to execute 'cc': No such file or directory
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    *********************************************************************************
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AeWcP9/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4tvJdQ-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-AeWcP9/lxml/

(Jeremy Howard) #34

As it mentions in the error ‘make sure the development packages of libxml2 and libxslt are installed’. They should already be part of anaconda - are you using our script to setup your AWS instance? It looks like you are either not using it, or are not connected to your instance, since your prompt above shows ‘kicho@PC08EYW5 ~’, but on our AWS setup your username would be ‘ubuntu’, not ‘kicho’.

On our AWS setup, you should not need to install any prerequisites in order to install kaggle-cli - just ‘pip install kaggle-cli’ should be it.


(garima.agarwal) #35

I was getting an error
ubuntu@ip-10-0-0-5:~/nbs$ kg download
Starting new HTTPS connection (1): www.kaggle.com
’NoneType’ object has no attribute ‘find_all’

There was a typo in the competition name. I was using dogs-vs-cats-redux-kernel-edition instead of dogs-vs-cats-redux-kernels-edition. That fixed it.

Thanks @jeremy for pointing out!


(Kicho) #36

Thank you, Jeremy! It works now. I accessed an AWS instance using our script and then pip install kaggle-cli worked. :slight_smile:


(Swathi Shyam Sunder) #37

http://forums.fast.ai/t/kaggle-questions/135/29?u=swathi.ssunder

@rachel - Thank you for pointing me to the evaluation page, where there is info about the submission s.
My question was about how to actually run the trained model on the test data? Was I right in trying to execute vgg.predict on the test data?
When I actually did this, the result was a tuple, which I converted to a numpy array through numpy.asarray and then wrote to a csv file.
I think there is something wrong in this.


(wendydherin) #38

Remember that the get_batches method shuffle has a default setting of True.


(Jeremy Howard) #39

@swathi.ssunder you’re on the right track, but you’ll need to spend some time looking at the data structure that’s returned from predict(), and figuring out how to modify it to be the correct structure to submit to kaggle. You’ll need to write some code to make these changes.

Give it your best shot, and if you get stuck, let us know exactly what you’ve tried, what you’ve found so far, and what it is that you need to do next and don’t know how to do.


(Swathi Shyam Sunder) #40

http://forums.fast.ai/t/kaggle-questions/135/39?u=swathi.ssunder

@jeremy - thanks for clarifying. sure, I shall do that.


#41

I used SciKit Learn’s StratifiedKFold and generating symlinks in python to dynamically generate the training and validation sets (and reset them, if necessary). I hope that’s ok.


(melissa.fabros) #42

thanks for the tip! also, are the file names the id names?


(Jeremy Howard) #43

@melissa.fabros yes that’s exactly right!


(bckenstler) #44

Just a heads up, batches.filenames will always list the directory order of the files, regardless of whether the generator is set to shuffle or not. So if you don’t shuffle, obtaining the image id’s is straightforward, but has anyone figured out how obtain the image id’s in the order that a shuffling iterator outputs them in? I don’t really see how to do it


(Jeremy Howard) #45

@bckenstler I think the correct answer is: don’t do that! :wink:


(sethiavivek2006) #46

@all I was trying with vgg.test for the dogs-vs-cats-redux-kernels-edition on my AWS p2 large instance. I have executed all the steps mentioned in the lesson2 lecture for creating separate directories. But when I perform the step as shown in the picture, my execution gets stuck( indicated by * preceding the statement) until I interrupt. Further steps do not work since the previous step was not executed properly. Any suggestion on how to solve this ?


(Jeremy Howard) #47

It’s likely to take up to 10 mins, since it has to run the forward pass of the neural net on all 12,500 images. If it’s taking longer, make sure that it’s using your GPU (check the result of the ‘import utils’ line)