I would like to know how to submit to Kaggle.
The sample_submission.csv file has 2 column headers id and label. I am supposing id refers to the image id. Does label refer to the probability of prediction or the category of the image?
Also, how do I generate a csv file from the output of vgg.predict?
I am new to Kaggle. So please excuse me if this is something very basic. I am also trying to find out myself.
@swathi.ssunder Looking at sample_submission.csv is a good way of seeing what the data should look like (if the values are probabilities or categories), but make sure that you are using sample_submission.csv for the right competition. Dogs-vs-cats and Dogs-vs-cats-redux-kernels-edition have different required submission formats, and part of the homework is figuring out how to process and format the output correctly.
You can also get info on submissions from looking at the Kaggle Evaluation page for that competition:
Dogs vs Cats Redux: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation
Dogs vs Cats: https://www.kaggle.com/c/dogs-vs-cats/details/evaluation
(again, note that they are different)
As for generating a csv (from a numpy array), can you tell me what you’ve tried searching for so far, or what methods you’ve already tried? I can give you a hint if you are stuck.
I have a Kaggle set up question. I tried pip install kaggle-cli in Cygwin at Windows 7, but I am not sure it was successful. I tried pip install python-lxml, but I am not sure it worked either. What shall I do?
kicho@PC08EYW5 ~
$ pip install kaggle-cli
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting kaggle-cli
Requirement already satisfied (use --upgrade to upgrade): cliff in /usr/lib/python2.7/site-packages (from kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): MechanicalSoup in /usr/lib/python2.7/site-packages (from kaggle-cli)
Collecting lxml (from kaggle-cli)
Downloading lxml-3.6.4.tar.gz (3.7MB)
100% |████████████████████████████████| 3.7MB 195kB/s
Collecting cssselect (from kaggle-cli)
Downloading cssselect-1.0.0-py2.py3-none-any.whl
Collecting configparser (from kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): cmd2>=0.6.7 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): PrettyTable<0.8,>=0.7 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): unicodecsv>=0.8.0; python_version < "3.0" in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): pbr>=1.6 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): stevedore>=1.16.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): PyYAML>=3.1.0 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=2.0.1 in /usr/lib/python2.7/site-packages (from cliff->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in /usr/lib/python2.7/site-packages (from MechanicalSoup->kaggle-cli)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.0 in /usr/lib/python2.7/site-packages (from MechanicalSoup->kaggle-cli)
Installing collected packages: lxml, cssselect, configparser, kaggle-cli
Running setup.py install for lxml ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xCaOTi/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QWptuE-record/install-record.txt --single-version-externally-managed --compile:
Building lxml version 3.6.4.
Building without Cython.
ERROR: /bin/sh: xslt-config: command not found
** make sure the development packages of libxml2 and libxslt are installed **
Using build configuration of libxslt
running install
running build
running build_py
creating build
creating build/lib.cygwin-2.6.0-x86_64-2.7
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/cssselect.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/doctestcompare.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/ElementInclude.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/pyclasslookup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/sax.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/_elementpath.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/clean.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/defs.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/diff.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/ElementSoup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/formfill.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/html5parser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/soupparser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/_diffcommand.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/_html5builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/_setmixin.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
copying src/lxml/isoschematron/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
copying src/lxml/lxml.etree.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/lxml.etree_api.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/includes/c14n.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/config.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/dtdvalid.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/etreepublic.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/htmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/relaxng.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/schematron.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/tree.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/uri.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xinclude.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xmlerror.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xmlschema.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xpath.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xslt.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/etree_defs.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/lxml-version.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
running build_ext
building 'lxml.etree' extension
creating build/temp.cygwin-2.6.0-x86_64-2.7
creating build/temp.cygwin-2.6.0-x86_64-2.7/src
creating build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml
gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/build=/usr/src/debug/python-2.7.12-1 -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/src/Python-2.7.12=/usr/src/debug/python-2.7.12-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml/lxml.etree.o -w
unable to execute 'gcc': No such file or directory
Compile failed: command 'gcc' failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitAWWi_U.c -o tmp/xmlXPathInitAWWi_U.o
unable to execute 'cc': No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-xCaOTi/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QWptuE-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-xCaOTi/lxml/
kicho@PC08EYW5 ~
$ pip install python-lxml
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting python-lxml
Could not find a version that satisfies the requirement python-lxml (from versions: )
No matching distribution found for python-lxml
@analytics20132333 it would be ‘pip install lxml’, not ‘pip install python-lxml’. Having said that, there’s really no reason to install kaggle cli on Windows - you can just use your browser to submit to Kaggle on Windows.
But I’m guessing this isn’t actually what you want - assuming that you’re doing your GPU training on Linux (i.e. an AWS instance), then you need to be installing kaggle CLI onto your GPU server, not into Cygwin. The purpose of Kaggle CLI is that it’s hard to navigate the Kaggle website and make a submission via your Linux server, so it’s generally easier to to submit through a command line interface (CLI). So you would type ‘pip install kaggle-cli’ into your bash shell on your server.
Does that make sense?
When working with the test set:
The get_batches function presumes the passed-in path will contain subdirectories, so you’ll want to create a subdirectory within your test folder (you can name it anything you want), and move all of the images into that subdirectory.
If you’re wondering how to get the filenames for the batched images, there is a .filenames method you can call…
Hi @jeremy! I think it makes sense. I was simply following the assignment instruction. I was at Cygwin level, but I tried again on my AWS instance after reading your comment. I got the same error after trying pip install lxml. Is there a way to make the command line interface installation possible on an AWS instance? Or should I use a browser in Windows?
kicho@PC08EYW5 ~
$ pip install lxml
The directory '/home/kicho/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/kicho/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
Downloading lxml-3.6.4.tar.gz (3.7MB)
100% |████████████████████████████████| 3.7MB 276kB/s
Installing collected packages: lxml
Running setup.py install for lxml ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AeWcP9/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4tvJdQ-record/install-record.txt --single-version-externally-managed --compile:
Building lxml version 3.6.4.
Building without Cython.
ERROR: /bin/sh: xslt-config: command not found
** make sure the development packages of libxml2 and libxslt are installed **
Using build configuration of libxslt
running install
running build
running build_py
creating build
creating build/lib.cygwin-2.6.0-x86_64-2.7
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/cssselect.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/doctestcompare.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/ElementInclude.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/pyclasslookup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/sax.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/_elementpath.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/clean.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/defs.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/diff.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/ElementSoup.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/formfill.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/html5parser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/soupparser.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/usedoctest.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/_diffcommand.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/_html5builder.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/_setmixin.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
copying src/lxml/html/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/html
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
copying src/lxml/isoschematron/__init__.py -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron
copying src/lxml/lxml.etree.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/lxml.etree_api.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml
copying src/lxml/includes/c14n.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/config.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/dtdvalid.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/etreepublic.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/htmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/relaxng.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/schematron.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/tree.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/uri.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xinclude.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xmlerror.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xmlparser.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xmlschema.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xpath.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/xslt.pxd -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/etree_defs.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
copying src/lxml/includes/lxml-version.h -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/includes
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/rng
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl
creating build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.cygwin-2.6.0-x86_64-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
running build_ext
building 'lxml.etree' extension
creating build/temp.cygwin-2.6.0-x86_64-2.7
creating build/temp.cygwin-2.6.0-x86_64-2.7/src
creating build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml
gcc -fno-strict-aliasing -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/build=/usr/src/debug/python-2.7.12-1 -fdebug-prefix-map=/usr/src/ports/python/python-2.7.12-1.x86_64/src/Python-2.7.12=/usr/src/debug/python-2.7.12-1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.cygwin-2.6.0-x86_64-2.7/src/lxml/lxml.etree.o -w
unable to execute 'gcc': No such file or directory
Compile failed: command 'gcc' failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitd1Hw8A.c -o tmp/xmlXPathInitd1Hw8A.o
unable to execute 'cc': No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AeWcP9/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4tvJdQ-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-AeWcP9/lxml/
As it mentions in the error ‘make sure the development packages of libxml2 and libxslt are installed’. They should already be part of anaconda - are you using our script to setup your AWS instance? It looks like you are either not using it, or are not connected to your instance, since your prompt above shows ‘kicho@PC08EYW5 ~’, but on our AWS setup your username would be ‘ubuntu’, not ‘kicho’.
On our AWS setup, you should not need to install any prerequisites in order to install kaggle-cli - just ‘pip install kaggle-cli’ should be it.
I was getting an error
ubuntu@ip-10-0-0-5:~/nbs$ kg download
Starting new HTTPS connection (1): www.kaggle.com
’NoneType’ object has no attribute ‘find_all’
There was a typo in the competition name. I was using dogs-vs-cats-redux-kernel-edition instead of dogs-vs-cats-redux-kernels-edition. That fixed it.
Thanks @jeremy for pointing out!
Thank you, Jeremy! It works now. I accessed an AWS instance using our script and then pip install kaggle-cli worked. 
http://forums.fast.ai/t/kaggle-questions/135/29?u=swathi.ssunder
@rachel - Thank you for pointing me to the evaluation page, where there is info about the submission s.
My question was about how to actually run the trained model on the test data? Was I right in trying to execute vgg.predict on the test data?
When I actually did this, the result was a tuple, which I converted to a numpy array through numpy.asarray and then wrote to a csv file.
I think there is something wrong in this.
Remember that the get_batches method shuffle has a default setting of True.
@swathi.ssunder you’re on the right track, but you’ll need to spend some time looking at the data structure that’s returned from predict(), and figuring out how to modify it to be the correct structure to submit to kaggle. You’ll need to write some code to make these changes.
Give it your best shot, and if you get stuck, let us know exactly what you’ve tried, what you’ve found so far, and what it is that you need to do next and don’t know how to do.
http://forums.fast.ai/t/kaggle-questions/135/39?u=swathi.ssunder
@jeremy - thanks for clarifying. sure, I shall do that.
I used SciKit Learn’s StratifiedKFold and generating symlinks in python to dynamically generate the training and validation sets (and reset them, if necessary). I hope that’s ok.
thanks for the tip! also, are the file names the id names?
Just a heads up, batches.filenames will always list the directory order of the files, regardless of whether the generator is set to shuffle or not. So if you don’t shuffle, obtaining the image id’s is straightforward, but has anyone figured out how obtain the image id’s in the order that a shuffling iterator outputs them in? I don’t really see how to do it
@all I was trying with vgg.test for the dogs-vs-cats-redux-kernels-edition on my AWS p2 large instance. I have executed all the steps mentioned in the lesson2 lecture for creating separate directories. But when I perform the step as shown in the picture, my execution gets stuck( indicated by * preceding the statement) until I interrupt. Further steps do not work since the previous step was not executed properly. Any suggestion on how to solve this ?
It’s likely to take up to 10 mins, since it has to run the forward pass of the neural net on all 12,500 images. If it’s taking longer, make sure that it’s using your GPU (check the result of the ‘import utils’ line)
