Skip to content

Commit ca6cb5a

Browse files
author
Matt Crane
committed
Following in the best traditions of Muphry's Law!
The scripts here didn't quite fully replicate the paper, so I've fixed the scripts, made them better, and added a README with all the juicy details on which scripts to run, to get which results, and why some things don't match the upstream repo
1 parent 20cc333 commit ca6cb5a

File tree

11 files changed

+380
-106
lines changed

11 files changed

+380
-106
lines changed

‎Dockerfile

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu14.04 as base
1212
COPY aquaint+wiki.txt.gz.ndim=50.bin /
1313

1414
RUN echo "13dc26ecb4455cf437e19b6dcf869867 *aquaint+wiki.txt.gz.ndim=50.bin" | md5sum -c - && \
15-
apt-get update --fix-missing && apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
15+
apt-get update --fix-missing && apt-get install -y wget unzip bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
1616
echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh && \
1717
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-x86_64.sh -O ~/miniconda.sh && \
1818
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
@@ -53,21 +53,24 @@ ENV OMP_NUM_THREADS=1
5353
ENV MKL_NUM_THREADS=1
5454

5555
#
56-
# And now get the repositories, with `castor` at a given sha sum
56+
# And finally the latest verison, see the README for why this is separate
5757
#
5858
FROM pytorch
5959

60-
ARG sha=cf0e269
61-
62-
RUN git clone https://github.com/castorini/castor /castorini/castor && \
63-
git clone https://github.com/castorini/data /castorini/data && \
64-
git -C /castorini/castor reset --hard ${sha} && \
65-
git -C /castorini/data reset --hard 42abddd && \
60+
COPY castor /castorini/castor
61+
RUN git clone https://github.com/castorini/data /castorini/data && \
62+
git -C /castorini/data reset --hard 6ed4084 && \
6663
mv /aquaint+wiki.txt.gz.ndim=50.bin /castorini/data/word2vec && \
6764
cd /castorini/data/TrecQA && \
6865
python parse.py && \
6966
python overlap_features.py && \
7067
python build_vocab.py && \
68+
cd /castorini/data/WikiQA && \
69+
unzip WikiQACorpus.zip && \
70+
python create-train-dev-test-data.py && \
71+
mv train train-all && \
72+
mv test raw-test && \
73+
mv dev raw-dev && \
7174
cd /castorini/castor/sm_cnn/trec_eval-8.0 && \
7275
make
7376

‎Dockerfile.change-sha

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#
2+
# First, get the nvidia image, which has the nvidia cuda drivers etc.
3+
#
4+
FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu14.04 as base
5+
6+
#
7+
# Install miniconda into it
8+
## Copied from continuumio/miniconda:4.3.14
9+
#
10+
11+
# Do this here, because it'll be cached
12+
COPY aquaint+wiki.txt.gz.ndim=50.bin /
13+
14+
RUN echo "13dc26ecb4455cf437e19b6dcf869867 *aquaint+wiki.txt.gz.ndim=50.bin" | md5sum -c - && \
15+
apt-get update --fix-missing && apt-get install -y wget unzip bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
16+
echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh && \
17+
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.3.14-Linux-x86_64.sh -O ~/miniconda.sh && \
18+
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
19+
rm ~/miniconda.sh && \
20+
apt-get install -y curl grep sed dpkg && \
21+
TINI_VERSION=`curl https://github.com/krallin/tini/releases/latest | grep -o "/v.*\"" | sed 's:^..\(.*\).$:\1:'` && \
22+
curl -L "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.deb" > tini.deb && \
23+
dpkg -i tini.deb && \
24+
rm tini.deb
25+
26+
ENV PATH /opt/conda/bin:$PATH
27+
28+
#
29+
# Then install pytorch, version with/without mkl
30+
## ORIGINALLY FROM continuumio/miniconda:4.3.14
31+
#
32+
FROM base as pytorch
33+
34+
ARG lib=mkl
35+
ARG pytorch=0.1.12
36+
37+
RUN apt install -y build-essential=11.6ubuntu6 && \
38+
apt-get clean && \
39+
export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" && \
40+
conda install -y conda==4.3.25 python==3.6.1 && \
41+
if [ "${lib}" = "mkl" ]; then \
42+
conda install -y mkl==2017.0.3 mkl-service==1.1.2; \
43+
else \
44+
conda install -y nomkl==1.0; \
45+
fi && \
46+
conda install -y numpy==1.13.1 pyyaml==3.12 setuptools==27.2.0 cmake==3.6.3 gcc==4.8.5 cffi==1.10.0 gensim==1.0.1 nltk==3.2.1 scikit-learn==0.18.1 pandas==0.20.3 && \
47+
wget https://github.com/pytorch/pytorch/archive/v${pytorch}.tar.gz && \
48+
tar xzf v${pytorch}.tar.gz && \
49+
cd pytorch-${pytorch} && \
50+
python setup.py install
51+
52+
ENV OMP_NUM_THREADS=1
53+
ENV MKL_NUM_THREADS=1
54+
55+
#
56+
# And now get the repositories, with `castor` at a given sha sum
57+
#
58+
FROM pytorch
59+
60+
ARG sha=cf0e269
61+
62+
RUN git clone https://github.com/snapbug/castor /castorini/castor && \
63+
git clone https://github.com/castorini/data /castorini/data && \
64+
git -C /castorini/castor reset --hard ${sha} && \
65+
git -C /castorini/data reset --hard 6ed4084 && \
66+
mv /aquaint+wiki.txt.gz.ndim=50.bin /castorini/data/word2vec && \
67+
cd /castorini/data/TrecQA && \
68+
python parse.py && \
69+
python overlap_features.py && \
70+
python build_vocab.py && \
71+
cd /castorini/data/WikiQA && \
72+
unzip WikiQACorpus.zip && \
73+
python create-train-dev-test-data.py && \
74+
mv train train-all && \
75+
mv test raw-test && \
76+
mv dev raw-dev && \
77+
cd /castorini/castor/sm_cnn/trec_eval-8.0 && \
78+
make
79+
80+
WORKDIR /castorini/castor/sm_cnn

‎README.md

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Questionable Answers
2+
3+
This repository contains everything required to completely replicate the results presented in:
4+
5+
Matt Crane. "Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results". In: Transactions of the Association for Computational Linguistics 6 (2018), pp. 241–252. url: https://transacl.org/ojs/index.php/tacl/article/view/1299.
6+
7+
## Status
8+
9+
Unfortunately, the upstream repository
10+
[castorini/castor](//github.com/castorini/castor) has diverged due to history
11+
rewriting changes, so the changesets don't match the official current
12+
repository.
13+
14+
Unfortunately this repository was not forked in time to capture the `cf0e269`
15+
SHA from the official repository before that repositories history was
16+
re-written. This means that if building from source, you'll have a _different_
17+
SHA which is used to build this image. The `setup.sh` script will make this
18+
change, the contents of which can be verified against [the official repository
19+
diff](//github.com/castorini/castor/commit/ed4dba249712e8bbaf5ed7c1486dff52b472daf4).
20+
21+
Running `setup.sh build` will build the docker images from the source,
22+
including making the un-captured change above, while `setup.sh pull` will pull
23+
the prebuilt docker images.
24+
25+
## Requirements
26+
27+
#### Running on GPU
28+
29+
`nvidia-docker` is required to run the GPU based experiments, and for these
30+
experiments version 1 was used. This has since been deprecated by nVidia in
31+
favour of version 2. The results _should_ be the same, but for guarantees
32+
[install version 1](//github.com/nvidia/nvidia-docker/wiki/Installation-(version-1.0)).
33+
34+
#### Building images
35+
36+
The embeddings used by the network should be downloaded from [Aliaksei Severyn's shared file
37+
(520MB)](//drive.google.com/folderview?id=0B-yipfgecoSBfkZlY2FFWEpDR3M4Qkw5U055MWJrenE5MTBFVXlpRnd0QjZaMDQxejh1cWs&usp=sharing),
38+
and placed in the working directory for this repository. The docker image
39+
builder will verify checksums to ensure that the same file is used.
40+
41+
#### Pulling images
42+
43+
All the docker images generated are available online to download/run without
44+
having to be built from scratch. These are listed [on Docker hub](//hub.docker.com/r/snapbug/qqa/tags/)
45+
46+
By default the `setup.sh` script if run with will pull _all_ the tagged images,
47+
this can take a substantial amount of disk space, even though they share a lot
48+
of commonality. If you only, for example, want to replicate the math library
49+
experiments, then manually pull the required images. Look at `run.sh` for which
50+
images are required for which experiments.
51+
52+
| Image | Figure/Table | Notes |
53+
|---------------|------------------|------------------------------------------------|
54+
| `sha-*` | Table 4 | See note above regarding `sha-cf0e269` |
55+
| `pytorch-*` | Table 5 | |
56+
| `*mkl` | Table 6 | |
57+
| `sha-cf0e269` | Table 7 | |
58+
| `sha-cf0e269` | Table 8 | |
59+
| `sha-cf0e269` | Figure 2 (left) | Just the CPU seeds |
60+
| `sha-cf0e269` | Figure 2 (right) | Just the GPU seeds |
61+
| `sha-cf0e269` | Figure 2 | Both CPU and GPU seeds |
62+
| | Figure 3 | Use the output from the logs of `run.sh seeds` |
63+
| | Table 9 | Use the output from the logs of `run.sh seeds` |
64+
65+
## Replication
66+
67+
`run.sh` will successfully replicate all the experiments in the paper using
68+
either the built docker images, or pulled docker images from `setup.sh`. It
69+
takes a single argument that specifies which experiments to run.
70+
71+
| Argument | Figure/Table | Notes |
72+
|-----------|------------------|------------------------------------------------|
73+
| all | | All of the experiments |
74+
| network | Table 4 | |
75+
| pytorch | Table 5 | |
76+
| mathlib | Table 6 | |
77+
| thread | Table 7 | |
78+
| gpu | Table 8 | |
79+
| seeds-cpu | Figure 2 (left) | Just the CPU seeds |
80+
| seeds-gpu | Figure 2 (right) | Just the GPU seeds |
81+
| seeds | Figure 2 | Both CPU and GPU seeds |
82+
| | Figure 3 | Use the output from the logs of `run.sh seeds` |
83+
| | Table 9 | Use the output from the logs of `run.sh seeds` |
84+
85+
**Log** files are generated in the form `qqa.[dataset].log.[experiment]`, at the
86+
end of training the network performs a feed-forward pass of the datasets, which
87+
is where the numbers for the paper are extracted.
88+
89+
**Model** files will be generated in the form:
90+
`qqa.[dataset].model.[experiment]`, to allow for feed-forward verification, or
91+
re-creation of the results without retraining the network. These models, in my
92+
experimentation, are reproducible across different hardware setups, although I
93+
would be interested in hearing of situations where they _aren't_.
94+
95+
## Issues
96+
97+
If you encounter any issues with the scripts etc. in this repository, then
98+
either file an issue on github, or email me.

‎gpu.sh

Lines changed: 0 additions & 16 deletions
This file was deleted.

‎network_version.sh

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎nomkl.sh

Lines changed: 0 additions & 5 deletions
This file was deleted.

‎pytorch_versions.sh

Lines changed: 0 additions & 13 deletions
This file was deleted.

0 commit comments

Comments
 (0)