Preparing for release 0.1

SeleniumRule is the public API. Provides:
- automatic setup and teardown of a WebDriver (WebDriverResource)
- retry logic in case of test failure, provided that the test is annotated with Flaky (RetryRule)

Leaving out for the moment:
- TakeScreenshotOnFailureRule
- TestLoggerRule
which need a TestReporterRule first.

Author: alb-i986

README.md

@@ -0,0 +1,103 @@
+# Selenium-JUnit4
+
+A light-weight Selenium testing framework providing a number of features in the form of
+[JUnit Rules](https://github.com/junit-team/junit4/wiki/Rules).
+
+
+## Motivation
+When starting a test automation project using Selenium, there are a few things that need to be
+implemented.
+In fact, there's a gap to fill between Selenium and the testing framework of choice.
+A classic example is the setup and tear down of a `WebDriver` for each test.
+Or to take a screenshot in case of test failure. Selenium provides a method for taking a screenshot,
+but it's up to the tester to call it when a test fails.
+All of these things are pretty much a must-have, so we all (testers) end up reinventing the wheel.
+
+The aim of this project is to fill the gap for projects using JUnit as the testing framework.
+The way it does is by exploiting [JUnit Rules](https://github.com/junit-team/junit4/wiki/Rules),
+which by the way allows for a *clean* design.
+
+
+## Features
+
+- Automatic setup and tear down of `WebDriver`'s
+- Retry flaky tests in case of failure
+
+Soon to be added:
+
+- Screenshot on test failure
+- HTML test reports (with screenshots on test failure)
+
+
+## Example of usage
+
+    public class MySeleniumTest {
+
+       @Rule
+       public final SeleniumRule seleniumRule = new SeleniumRule(new ChromeDriverFactory())
+           .toRetryFlakyTestsOnFailure(2); // retry each test max 2 times (max 3 executions in total)
+
+       protected final WebDriver driver() {
+           return seleniumRule.getDriver();
+       }
+
+       @Test
+       @Flaky // without this, the test will *not* be re-tried in case of failure, even though SeleniumRule was configured to retry
+       public void myFlakyTest() {
+           throw new RuntimeException("flaky test failure");
+       }
+
+       @Test
+       public void myStableTest() {
+           driver().get("http://www.google.com");
+           driver().findElement(By.name("q")).sendKeys("selenium-junit" + Keys.ENTER);
+           new WebDriverWait(driver(), 5).until(ExpectedConditions.titleContains("selenium-junit"));
+       }
+
+       private static class ChromeDriverFactory implements WebDriverFactory {
+           @Override
+           public WebDriver create() {
+               return new ChromeDriver();
+           }
+       }
+    }
+
+In this example we have a test class with two tests, `myStableTest` and `myFlakyTest`,
+and one Rule, a `SeleniumRule`.
+
+In `myStableTest` we are using the driver returned by `seleniumRule.getDriver()`
+(which, for convenience, is wrapped in the method `driver()`)
+to interact with the web UI of the System Under Test.
+Thanks to `SeleniumRule`, the driver has already been initialized
+(in this example, it will be a local instance of the browser Chrome),
+and it will also be torn down automatically as soon as the test finishes.
+
+`myFlakyTest` is simulating a flaky test by always throwing an exception.
+Since the test is annotated with `@Flaky`, and `SeleniumRule` is configured to retry flaky tests
+on failure (`.toRetryFlakyTestsOnFailure(2)`), then `myFlakyTest` will be executed 3 times
+(1 standard execution + 2 retries, as per configuration).
+
+All of the details regarding the setup/teardown and the retry logic are hidden behind three lines:
+
+    @Rule
+    public final SeleniumRule seleniumRule = new SeleniumRule(new ChromeDriverFactory())
+       .toRetryFlakyTestsOnFailure(2); // retry each test max 2 times (max 3 executions in total)
+
+`SeleniumRule` is the one and only class being part of the public API of this project.
+It can be configured so that users can get only the features they need.
+The most minimal configuration includes only the setup and tear down of a WebDriver:
+
+    @Rule
+    public final SeleniumRule seleniumRule = new SeleniumRule(new ChromeDriverFactory());
+
+Please see the javadoc of `SeleniumRule` for an up-to-date example of usage.
+
+
+## Internals
+
+Internally, `SeleniumRule` is not a monolithic Rule implementing all of the features.
+Rather, each feature is implemented by a `TestRule` on its own.
+`SeleniumRule` simply acts as a collector (a `RuleChain`) of the rules configured:
+
+- makes sure the sub-rules are run in the correct order
+- provides clients with a configuration API to activate only the features, aka sub-rules, needed

pom.xml

@@ -6,7 +6,7 @@
 
     <groupId>me.alb-i986.selenium</groupId>
     <artifactId>selenium-junit4</artifactId>
-    <version>1.0-SNAPSHOT</version>
+    <version>0.1-SNAPSHOT</version>
     <description>A lightweight framework for writing Selenium tests with JUnit, providing a number of TestRule's</description>
     <url>https://github.com/alb-i986/selenium-junit4</url>
 
@@ -125,12 +125,6 @@
             <version>1.10.19</version>
             <scope>test</scope>
         </dependency>
-        <dependency>
-            <groupId>org.seleniumhq.selenium</groupId>
-            <artifactId>selenium-chrome-driver</artifactId>
-            <version>${selenium.version}</version>
-            <scope>test</scope>
-        </dependency>
     </dependencies>
 
     <distributionManagement>

src/main/java/me/alb_i986/selenium/WebDriverProvider.java

@@ -0,0 +1,15 @@
+package me.alb_i986.selenium;
+
+import org.openqa.selenium.WebDriver;
+
+/**
+ * Provides initialized {@link WebDriver} instances.
+ */
+public interface WebDriverProvider {
+
+    /**
+     * @return a non-null driver
+     * @throws IllegalStateException if the driver to be returned is null
+     */
+    WebDriver getDriver() throws IllegalStateException;
+}

src/main/java/me/alb_i986/selenium/junit/rules/DriverServiceResource.java

@@ -4,11 +4,33 @@
 import org.openqa.selenium.remote.service.DriverService;
 
 /**
- * A {@link org.junit.rules.TestRule} managing {@link DriverService} instances.
+ * A {@link org.junit.rules.TestRule} managing a {@link DriverService},
+ * i.e. starting and stopping it.
  * <p>
- * The {@link DriverService} is started on test start, and stopped on test termination.
+ * This rule should be used as a {@link org.junit.ClassRule}.
  * <p>
- * This rule is typically used as a {@link org.junit.ClassRule}.
+ * Example of usage:
+ * <pre>
+ * public class MyTest {
+ *
+ *     protected static final ChromeDriverService CHROME_DRIVER_SERVICE = ChromeDriverService.createDefaultService();
+ *
+ *     &#064;ClassRule
+ *     public static final DriverServiceResource DRIVER_SERVICE_RESOURCE = new DriverServiceResource(CHROME_DRIVER_SERVICE);
+ *
+ *     &#064;Test
+ *     public void firstTest() {
+ *         WebDriver driver = new ChromeDriver(CHROME_DRIVER_SERVICE);
+ *         [..]
+ *     }
+ *
+ *     &#064;Test
+ *     public void secondTest() {
+ *         WebDriver driver = new ChromeDriver(CHROME_DRIVER_SERVICE);
+ *         [..]
+ *     }
+ * }
+ * </pre>
  */
 public class DriverServiceResource extends ExternalResource {
 

src/main/java/me/alb_i986/selenium/junit/rules/Flaky.java

@@ -0,0 +1,15 @@
+package me.alb_i986.selenium.junit.rules;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * A test marked as {@code Flaky} will be retried in case of failure
+ * by {@link RetryRule}, if configured.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.METHOD)
+public @interface Flaky {
+}

src/main/java/me/alb_i986/selenium/junit/rules/RetryException.java

@@ -0,0 +1,45 @@
+package me.alb_i986.selenium.junit.rules;
+
+import org.junit.runner.Description;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Thrown by {@link RetryRule} when a test is retried and never passes.
+ * <p>
+ * Its message lists all of the test failures which occurred while retrying,
+ * with the stacktrace.
+ */
+public class RetryException extends RuntimeException {
+
+    public RetryException(Description description, Throwable... failures) {
+        this(description, Arrays.asList(failures));
+    }
+
+    public RetryException(Description description, List<Throwable> failures) {
+        super(createMessage(description, failures));
+    }
+
+    private static String createMessage(Description description, List<Throwable> failures) {
+        if (failures.isEmpty()) {
+            throw new IllegalArgumentException("The list of failures must not be empty");
+        }
+        StringBuilder sb = new StringBuilder();
+        sb.append(String.format("Flaky test '%s' failed %d times:\n", description.getDisplayName(), failures.size()));
+        int i = 1;
+        for (Throwable failure : failures) {
+            sb.append(String.format("\n  %d. %s", i++, getStackTrace(failure)));
+        }
+        return sb.toString();
+    }
+
+    // TODO with JUnit 4.13, we'll be able to use Throwables.getStacktrace() instead of this
+    private static String getStackTrace(Throwable throwable) {
+        StringWriter sw = new StringWriter();
+        throwable.printStackTrace(new PrintWriter(sw, true));
+        return sw.toString();
+    }
+}

src/main/java/me/alb_i986/selenium/junit/rules/RetryRule.java

@@ -0,0 +1,105 @@
+package me.alb_i986.selenium.junit.rules;
+
+import org.junit.internal.AssumptionViolatedException;
+import org.junit.rules.TestRule;
+import org.junit.runner.Description;
+import org.junit.runners.model.Statement;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * A rule which, in case of failure, re-runs a test annotated with {@link Flaky} until it passes,
+ * for max {@code n} times (max {@code n+1} executions in total).
+ * <p>
+ * If a test doesn't pass after {@code n+1} executions, a {@link RetryException} is thrown,
+ * whose message contains all of the failures occurred.
+ * <p>
+ * A test is retried <i>unless</i>:
+ * <ul>
+ *     <li>The test is <i>not</i> annotated with {@link Flaky}</li>
+ *     <li>The test throws {@link AssumptionViolatedException}</li>
+ *     <li>The test throws an {@link Error}, excluding {@link AssertionError},
+ *     so that OOM errors are always propagated ASAP</li>
+ *     <li>The configured retry times is 0 (i.e. {@code new RetryRule(0)}),
+ *     in which case any test will be executed only once,
+ *     as if there was no {@link RetryRule} in place</li>
+ * </ul>
+ * <h3>Example</h3>
+ * <pre>
+ * &#064;Rule TestRule retryRule = new RetryRule(1);
+ * </pre>
+ *
+ * Given the rule above, this is what happens for each test run:
+ * <ul>
+ *     <li>If the first execution is successful, that's it</li>
+ *     <li>Otherwise, the test will be run once again</li>
+ *     <li>If the second execution is still not successful,
+ *     then a {@link RetryException} will be thrown,
+ *     with the two test failures occurred attached to the exception message</li>
+ * </ul>
+ */
+public class RetryRule implements TestRule {
+
+    private final int retryTimes;
+
+    /**
+     * @param retries how many times to retry, <i>after the first attempt</i>.
+     *                For example, {@code new RetryRule(1)} will run a test max 2 times.
+     */
+    public RetryRule(int retries) {
+        if (retries < 0) {
+            throw new IllegalArgumentException("The number of retries needs to be an integer >= 0" +
+                    " but was: " + retries);
+        }
+        this.retryTimes = retries;
+    }
+
+    @Override
+    public Statement apply(final Statement base, final Description description) {
+        return new Statement() {
+            @Override
+            public void evaluate() throws Throwable {
+                // DO NOT retry if the test is not marked as flaky,
+                // or this rule has been configured to retry for 0 times
+                if (retryTimes == 0 || !isTestFlaky()) {
+                    base.evaluate();
+                    return;
+                }
+
+                List<Throwable> failures = new ArrayList<>();
+                int i;
+                for (i = 0; i < (retryTimes + 1); i++) {
+                    try {
+                        base.evaluate();
+                        break;
+                    } catch (AssertionError e) {
+                        failures.add(e);
+                    } catch (Error error) {
+                        throw error;
+                    } catch (AssumptionViolatedException e) {
+                        throw e;
+                    } catch (Throwable t) {
+                        failures.add(t);
+                    }
+
+                    // TODO report that it's going to retry
+                    // System.err.println(description.getDisplayName() +
+                    //             ": Failed, " + i + "retries remain");
+                }
+
+                if (!failures.isEmpty()) { // the test was retried
+                    if (i == retryTimes + 1) { // the test failed all the times
+                        throw new RetryException(description, failures);
+                    } else { // the test failed a few times but passed in the end
+                        // TODO report the first (retryTimes - i) failures
+                    }
+                }
+            }
+
+            private boolean isTestFlaky() {
+                return description.getAnnotation(Flaky.class) != null;
+            }
+        };
+    }
+}