| layout | title |
|---|---|
global |
Spark Hadoop3 Integration Tests |
As mocking of an external systems (like AWS S3) is not always perfect the unit testing should be
extended with integration testing. This is why the build profile integration-test has been
introduced here. When it is given (-Pintegration-test) for testing then only those tests are
executed where the org.apache.spark.internal.io.cloud.IntegrationTestSuite tag is used.
One example is AwsS3AbortableStreamBasedCheckpointFileManagerSuite.
Integration tests will have some extra configurations for example selecting the external system to
run the test against. Those configs are passed as environment variables and the existence of these
variables must be checked by the test.
Like for AwsS3AbortableStreamBasedCheckpointFileManagerSuite the S3 bucket used for testing
is passed in the S3A_PATH and the credentials to access AWS S3 are AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY (in addition you can define optional AWS_SESSION_TOKEN and AWS_ENDPOINT_URL too).