Update datasets

openphilanthropy · Sep 13, 2018 · 10afc22 · 10afc22
1 parent 3b32845
commit 10afc22
Show file tree

Hide file tree

Showing 2 changed files with 13 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -17,17 +17,13 @@ We include three attacks in [the warm-up to the contest](warmup.md):
 The top few distinct models for each dataset are shown below.  You can see all submissions in [the full scoreboard](scoreboard.md). 
 
 #### Two-Class MNIST dataset
-![tcu_mnist_example 2x](https://user-images.githubusercontent.com/306655/45503359-7e04d700-b73b-11e8-87c1-9aa4167e7291.png)
-
 | Defense               | Submitted by  | Clean data | Spatial grid attack | SPSA attack | L2-ball attack |  Submission Date |
 | --------------------- | ------------- | ------------ |------------ |--------------- |--------------- | --------------- |
 | [MadryPGD LeNet Baseline](#)  |  Google Brain |    100.0%    |      ??    |     ??   |     ??     |  Aug 28th, 2018 |
 | [Undefended LeNet Baseline](#)   |  Google Brain   |    100.0%    |     ??    |     ??    |     ??     |  Aug 27th, 2018 |
 All percentages above correspond to the model's accuracy at 80% coverage.
 
 #### Bird or Bicycle dataset
-![bob_example 2x](https://user-images.githubusercontent.com/306655/45503358-7e04d700-b73b-11e8-895e-dec5bd2d7394.png)
-
 | Defense               | Submitted by  | Clean data | Spatial grid attack | SPSA attack | L2-ball attack |  Submission Date |
 | --------------------- | ------------- | ------------| ------------ |--------------- |--------------- | --------------- |
 | [Pytorch ResNet <br>(via bird-or-bicycle extras)](unrestricted_advex/undefended_pytorch_resnet)  |  Google Brain |    99.0%    |     45.2%   | 12.8%   |     ??     |  Sept 13th, 2018 |

diff --git a/warmup.md b/warmup.md
@@ -1,5 +1,18 @@
 # Warm-up to the Unrestricted Advex Contest
 
+We currently have three attacks that we're evaluating during the warmup
+
+- 1000 Linfinity-ball adversarial examples generated by SPSA
+- 1000 spatial adversarial examples (via grid search)
+- 100 L2-ball adversarial examples generated by a decision-only attack
+
+We have leaderboards for two datasets. See some samples below:
+
+#### Two-Class MNIST dataset
+![tcu_mnist_example_wide 2x](https://user-images.githubusercontent.com/306655/45507996-726bdd00-b748-11e8-90ad-ea701e63f9a2.png)
+
+#### Bird or Bicycle dataset 
+![bob_example_wide 2x](https://user-images.githubusercontent.com/306655/45507995-726bdd00-b748-11e8-9074-d93e257ea1a0.png)
 
 ### Implementing a defense