Tests are horribly blippy

[rclark]

Its almost impossible to push a commit and get a green light from Travis without mashing the retry button several times. In fact, its almost impossible to pass tests if you run them locally without trying a few times.

This may have to do with eventual consistency on S3, or just poorly designed tests. I'll try to identify some of the primary offending tests.

[mapsam]

Trying to overcome this by mocking s3 using @mapbox/mock-aws-sdk-js, but running into a situation where the usage of AWS.S3() in dependencies doesn't seem to be mocked, therefore running into credential errors for buckets that don't exist. Maybe mocking isn't the right way to do this?

[rclark]

The easiest way for @mapbox/mock-aws-sdk-js to not-mock your AWS SDK clients is if the client is defined in code before your test script initiates the mock.

If you're testing this script:

const AWS = require('aws-sdk');

const client = new AWS.S3();

module.exports = function() {
  // stuff you want to test
};

Then the client will be created un-mocked. Get around this by defining the client in the function itself:

const AWS = require('aws-sdk');

module.exports = function() {
  const client = new AWS.S3();
  // stuff you want to test
};

[springmeyer]

@millzpaugh is working on fixing these blipping tests over at #120. @millzpaugh could you log here (for the record) a quick summary of the types of failing tests your PR will fix?

[millzpaugh]

@springmeyer and I paired on fixes to address the flakiness of tests in mapbox tile copy.

We found two tests that were inconsistently failing and addressed the following issues.

Removed tile count test for out of bounds errors
Tile counts related to the copy coordinates exceed spherical mercator tests were inconsistently failing. Because of the asynchronicity needed to render new tiles (multiple threads are created, and the max thread count is directly related to the operating system's CPU count), tiles can be stored on S3 before an error surfaces related to coordinates being out of bounds. Unless the first tile rendered was the tile containing an out of bounds error, one or more tiles could successfully render before the error was caught. Thus, checking a tile count on S3 caused errors in tests that don't actually test the condition we want to know is working properly - whether or not a tileset errors if a tile has an out of bounds coordinate. We therefore altered the tile count test to check for the appropriate out of bounds error.

Changed stdout length check to stdout log check
These tests also fail because of the asynchronicity needed to render a new tile. Since the processing time of copying a file via process-stream is set log progress of the job every 100 ms, we cannot say with certainty that the file will run through progress-stream X number of times and match that number with logs in stdout. However, we do know that there will be at least one log related to the progress of a tile being copied. Therefore, we changed the test to reflect this certainty that there will be a recorded log and test that stdout logs something.