Merge branch 'master' of github.com:google/unrestricted-adversarial-e…

…xamples

README.md

@@ -1,4 +1,4 @@
-# Unrestricted Adversarial Examples Challenge
+# Unrestricted Adversarial Examples Challenge [![Build Status](https://travis-ci.org/google/unrestricted-adversarial-examples.svg?branch=master)](https://travis-ci.org/google/unrestricted-adversarial-examples)
 
 In the Unrestricted Adversarial Examples Challenge, attackers submit arbitrary adversarial inputs, and defenders are expected to assign low confidence to difficult inputs while retaining high confidence and accuracy on a clean, unambiguous test set. You can learn more about the motivation and structure of the contest in [our recent paper](https://drive.google.com/open?id=1T0yiu9LPv_Qh-qYhYFLj9dxjnkca8fkG)
 
@@ -10,22 +10,22 @@ This repository contains code for [the warm-up to the challenge](warmup.md), as
 ### <a name="leaderboard"></a>Leaderboard for the warm-up to the contest
 We include three attacks in [the warm-up to the contest](warmup.md):
 
-- 1000 Linfinity-ball adversarial examples generated by SPSA
-- 1000 spatial adversarial examples (via grid search)
-- 100 L2-ball adversarial examples generated by a decision-only attack
+- 1000 Linfinity-ball adversarial examples generated by [SPSA](https://arxiv.org/pdf/1802.05666.pdf)
+- 1000 spatial adversarial examples [(via grid search)](https://arxiv.org/abs/1712.02779)
+- 100 L2-ball adversarial examples generated by the [Boundary attack](https://arxiv.org/abs/1712.04248)
 
 The top few distinct models for each dataset are shown below.  You can see all submissions in [the full scoreboard](scoreboard.md). 
 
 #### Two-Class MNIST dataset
-| Defense               | Submitted by  | Clean data | Spatial grid attack | SPSA attack | L2-ball attack |  Submission Date |
+| Defense               | Submitted by  | Clean data | Spatial grid attack | SPSA attack | Boundary attack |  Submission Date |
 | --------------------- | ------------- | ------------ |------------ |--------------- |--------------- | --------------- |
-| [MadryPGD LeNet Baseline](#)  |  Google Brain |    100.0%    |      ??    |     ??   |     ??     |  Aug 28th, 2018 |
-| [Undefended LeNet Baseline](#)   |  Google Brain   |    100.0%    |     ??    |     ??    |     ??     |  Aug 27th, 2018 |
+| [MadryPGD LeNet Baseline](#)  |  Google Brain |    100.0%    |      0%    |     0%   |     0%     |  Sept 14th, 2018 |
+| [Undefended LeNet Baseline](#)   |  Google Brain   |    100.0%    |     0%    |     19.6%    |     0%     |  Sept 14th, 2018 |
 
 All percentages above correspond to the model's accuracy at 80% coverage.
 
 #### Bird or Bicycle dataset
-| Defense               | Submitted by  | Clean data | Spatial grid attack | SPSA attack | L2-ball attack |  Submission Date |
+| Defense               | Submitted by  | Clean data | Spatial grid attack | SPSA attack | Boundary attack |  Submission Date |
 | --------------------- | ------------- | ------------| ------------ |--------------- |--------------- | --------------- |
 | [Pytorch ResNet <br>(via bird-or-bicycle extras)](unrestricted_advex/undefended_pytorch_resnet)  |  Google Brain |    99.0%    |     45.2%   | 12.8%   |     ??     |  Sept 13th, 2018 |
 | [Keras ResNet <br>(via ImageNet)](unrestricted_advex/undefended_keras_resnet)   |  Google Brain   |    99.5%    |     ??    |     ??    |     ??     |  In progress |

contest_proposal.md

@@ -310,10 +310,12 @@ label the pixels of the bird/bicycle.
 (It is okay if the object is a photorealistic rendering of a bird/bicycle.)
 
 
-*The image is determined to be unambiguous ONLY IF all taskers answered
-  "Definitely yes" to one class, "Definitely no" to the other class, the
-  largest object is at least half of the image, is not truncated, is not occluded,
-  and is not a depiction of any sort.*
+### Criteria for a valid unambiguous bird-or-bicycle image
+The image is determined to be valid only if all of the following hold true:
+
+1. all taskers answered "Definitely yes" to one class, "Definitely no" to the other class
+2. the largest object is at least 25% of the image (as per the smallest bounding box by any tasker)
+3. the image is not truncated, is not occluded, and is not a depiction of any sort.
 
 
 # Additional Contest Mechanics FAQ

unrestricted-advex/unrestricted_advex/mnist_baselines/train_two_class_mnist.py

@@ -12,7 +12,7 @@
 
 
 def main(_):
-  assert train_mode in ['vanilla', 'adversarial']
+  assert FLAGS.train_mode in ['vanilla', 'adversarial']
   mnist = mnist_utils.mnist_dataset(one_hot=True)
   next_batch_fn = lambda: mnist.train.next_batch(FLAGS.batch_size)
   mnist_utils.train_mnist(FLAGS.model_dir, next_batch_fn,