Merge pull request #205 from nateprewitt/pre-commit

Add pre-commit config

.github/workflows/lint.yml

@@ -0,0 +1,19 @@
+name: Lint code
+
+on:
+  push:
+  pull_request:
+    branches-ignore: [ master ]
+
+jobs:
+  lint:
+    runs-on: ubuntu-20.04
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python 3.9
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.9
+    - name: Run pre-commit
+      uses: pre-commit/action@v2.0.0

.pre-commit-config.yaml

@@ -0,0 +1,12 @@
+exclude: ^(.changes/|CHANGELOG.rst)
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.0.1
+    hooks:
+    -   id: check-yaml
+    -   id: end-of-file-fixer
+    -   id: trailing-whitespace
+-   repo: https://github.com/pycqa/flake8
+    rev: 3.9.2
+    hooks:
+    -   id: flake8

ACCEPTANCE_TESTS.rst

@@ -95,7 +95,7 @@ Retry Logic
 ~~~~~~~~~~~
 * [x] Retry on connection related errors when downloading data
 * [ ] Compare MD5 to ``ETag`` and retry for mismatches if all following scenarios are met:
-      
+
   * If MD5 is available
   * Response does not have a ``ServerSideEncryption`` header equal to ``aws:kms``
   * Response does not have ``SSECustomerAlgorithm``
@@ -148,7 +148,7 @@ Validation
 Extra Parameters
 ~~~~~~~~~~~~~~~~
 * [ ] Copy multipart and nonmultipart file with any of the following properties:
-  
+
   * [x] ACL's
   * [x] CacheControl
   * [x] ContentDisposition

CHANGELOG.rst

@@ -182,4 +182,3 @@ CHANGELOG
 =====
 
 * feature:manager: Add boto3 s3 transfer logic to package. (`issue 2 <https://github.com/boto/s3transfer/pull/2>`__)
-

CONTRIBUTING.rst

@@ -0,0 +1,56 @@
+Contributing Code
+-----------------
+A good pull request:
+
+-  Is clear.
+-  Works across all supported versions of Python.
+-  Follows the existing style of the code base (see Codestyle section).
+-  Has comments included as needed.
+
+-  A test case that demonstrates the previous flaw that now passes with
+   the included patch, or demonstrates the newly added feature.
+-  If it adds/changes a public API, it must also include documentation
+   for those changes.
+-  Must be appropriately licensed (Apache 2.0).
+
+Reporting An Issue/Feature
+--------------------------
+First, check to see if there's an existing
+`issue <https://github.com/boto/botocore/issues>`__/`pull requests <https://github.com/boto/botocore/pulls>`__ for the bug/feature.
+
+If there isn't an existing issue there, please file an issue *first*. The
+ideal report includes:
+
+-  A description of the problem/suggestion.
+-  How to recreate the bug.
+-  If relevant, including the versions of your:
+
+   -  Python interpreter
+   -  s3transfer
+   -  Optionally of the other dependencies involved (e.g. Botocore)
+
+-  If possible, create a pull request with a (failing) test case
+   demonstrating what's wrong. This makes the process for fixing bugs
+   quicker & gets issues resolved sooner.
+
+Codestyle
+---------
+This project uses flake8 to enforce codstyle requirements. We've codified this
+process using a tool called `pre-commit <https://pre-commit.com/>`__. pre-commit
+allows us to specify a config file with all tools required for code linting,
+and surfaces either a git commit hook, or single command, for enforcing these.
+
+To validate your PR prior to publishing, you can use the following
+`installation guide <https://pre-commit.com/#install>`__ to setup pre-commit.
+
+If you don't want to use the git commit hook, you can run the below command
+to automatically perform the codestyle validation:
+
+.. code-block:: bash
+
+    $ pre-commit run
+
+This will automatically perform simple updates (such as white space clean up)
+and provide a list of any failing flake8 checks. After these are addressed,
+you can commit the changes prior to publishing the PR.
+These checks are also included in our CI setup under the "Lint" workflow which will provide output on Github for anything missed locally.

LICENSE.txt

@@ -200,4 +200,3 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-

requirements-dev-lock.txt

@@ -12,6 +12,14 @@ attrs==21.2.0 \
     --hash=sha256:149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1 \
     --hash=sha256:ef6aaac3ca6cd92904cdd0d83f629a15f18053ec84e6432106f7a4d04ae4f5fb
     # via pytest
+backports.entry-points-selectable==1.1.0 \
+    --hash=sha256:988468260ec1c196dab6ae1149260e2f5472c9110334e5d51adcb77867361f6a \
+    --hash=sha256:a6d9a871cde5e15b4c4a53e3d43ba890cc6861ec1332c9c2428c92f977192acc
+    # via virtualenv
+cfgv==3.3.1 \
+    --hash=sha256:c6a0883f3917a037485059700b9e75da2464e6c27051014ad85ba6aaa5884426 \
+    --hash=sha256:f5a830efb9ce7a445376bb66ec94c638a9787422f96264c98edc6bdeed8ab736
+    # via pre-commit
 colorama==0.4.4 \
     --hash=sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b \
     --hash=sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2
@@ -72,16 +80,40 @@ coverage==5.5 \
     # via
     #   -r requirements-dev.txt
     #   pytest-cov
+distlib==0.3.3 \
+    --hash=sha256:c8b54e8454e5bf6237cc84c20e8264c3e991e824ef27e8f1e81049867d861e31 \
+    --hash=sha256:d982d0751ff6eaaab5e2ec8e691d949ee80eddf01a62eaa96ddb11531fe16b05
+    # via virtualenv
+filelock==3.3.0 \
+    --hash=sha256:8c7eab13dc442dc249e95158bcc12dec724465919bdc9831fdbf0660f03d1785 \
+    --hash=sha256:bbc6a0382fe8ec4744ecdf6683a2e07f65eb10ff1aff53fc02a202565446cde0
+    # via virtualenv
+identify==2.3.0 \
+    --hash=sha256:d1e82c83d063571bb88087676f81261a4eae913c492dafde184067c584bc7c05 \
+    --hash=sha256:fd08c97f23ceee72784081f1ce5125c8f53a02d3f2716dde79a6ab8f1039fea5
+    # via pre-commit
 importlib-metadata==4.8.1 \
     --hash=sha256:b618b6d2d5ffa2f16add5697cf57a46c76a56229b0ed1c438322e4e95645bd15 \
     --hash=sha256:f284b3e11256ad1e5d03ab86bb2ccd6f5339688ff17a4d797a0fe7df326f23b1
     # via
     #   pluggy
+    #   pre-commit
     #   pytest
+    #   virtualenv
+importlib-resources==5.2.2 \
+    --hash=sha256:2480d8e07d1890056cb53c96e3de44fead9c62f2ba949b0f2e4c4345f4afa977 \
+    --hash=sha256:a65882a4d0fe5fbf702273456ba2ce74fe44892c25e42e057aca526b702a6d4b
+    # via
+    #   pre-commit
+    #   virtualenv
 iniconfig==1.1.1 \
     --hash=sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3 \
     --hash=sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32
     # via pytest
+nodeenv==1.6.0 \
+    --hash=sha256:3ef13ff90291ba2a4a7a4ff9a979b63ffdd00a464dbe04acf0ea6471517a4c2b \
+    --hash=sha256:621e6b7076565ddcacd2db0294c0381e01fd28945ab36bcf00f41c5daf63bef7
+    # via pre-commit
 nose==1.3.7 \
     --hash=sha256:9ff7c6cc443f8c51994b34a667bbcf45afd6d945be7477b52e97516fd17c53ac \
     --hash=sha256:dadcddc0aefbf99eea214e0f1232b94f2fa9bd98fa8353711dacb112bfcbbb2a \
@@ -91,10 +123,18 @@ packaging==21.0 \
     --hash=sha256:7dc96269f53a4ccec5c0670940a4281106dd0bb343f47b7471f779df49c2fbe7 \
     --hash=sha256:c86254f9220d55e31cc94d69bade760f0847da8000def4dfe1c6b872fd14ff14
     # via pytest
+platformdirs==2.4.0 \
+    --hash=sha256:367a5e80b3d04d2428ffa76d33f124cf11e8fff2acdaa9b43d545f5c7d661ef2 \
+    --hash=sha256:8868bbe3c3c80d42f20156f22e7131d2fb321f5bc86a2a345375c6481a67021d
+    # via virtualenv
 pluggy==1.0.0 \
     --hash=sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159 \
     --hash=sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3
     # via pytest
+pre-commit==2.15.0 \
+    --hash=sha256:3c25add78dbdfb6a28a651780d5c311ac40dd17f160eb3954a0c59da40a505a7 \
+    --hash=sha256:a4ed01000afcb484d9eb8d504272e642c4c4099bbad3a6b27e519bd6a3e928a6
+    # via -r requirements-dev.txt
 psutil==4.4.2 \
     --hash=sha256:10fbb631142a3200623f4ab49f8bf82c32b79b8fe179f6056d01da3dfc589da1 \
     --hash=sha256:15aba78f0262d7839702913f5d2ce1e97c89e31456bb26da1a5f9f7d7fe6d336 \
@@ -124,25 +164,67 @@ pytest==6.2.5 \
     # via
     #   -r requirements-dev.txt
     #   pytest-cov
+pyyaml==5.4.1 \
+    --hash=sha256:08682f6b72c722394747bddaf0aa62277e02557c0fd1c42cb853016a38f8dedf \
+    --hash=sha256:0f5f5786c0e09baddcd8b4b45f20a7b5d61a7e7e99846e3c799b05c7c53fa696 \
+    --hash=sha256:129def1b7c1bf22faffd67b8f3724645203b79d8f4cc81f674654d9902cb4393 \
+    --hash=sha256:294db365efa064d00b8d1ef65d8ea2c3426ac366c0c4368d930bf1c5fb497f77 \
+    --hash=sha256:3b2b1824fe7112845700f815ff6a489360226a5609b96ec2190a45e62a9fc922 \
+    --hash=sha256:3bd0e463264cf257d1ffd2e40223b197271046d09dadf73a0fe82b9c1fc385a5 \
+    --hash=sha256:4465124ef1b18d9ace298060f4eccc64b0850899ac4ac53294547536533800c8 \
+    --hash=sha256:49d4cdd9065b9b6e206d0595fee27a96b5dd22618e7520c33204a4a3239d5b10 \
+    --hash=sha256:4e0583d24c881e14342eaf4ec5fbc97f934b999a6828693a99157fde912540cc \
+    --hash=sha256:5accb17103e43963b80e6f837831f38d314a0495500067cb25afab2e8d7a4018 \
+    --hash=sha256:607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e \
+    --hash=sha256:6c78645d400265a062508ae399b60b8c167bf003db364ecb26dcab2bda048253 \
+    --hash=sha256:72a01f726a9c7851ca9bfad6fd09ca4e090a023c00945ea05ba1638c09dc3347 \
+    --hash=sha256:74c1485f7707cf707a7aef42ef6322b8f97921bd89be2ab6317fd782c2d53183 \
+    --hash=sha256:895f61ef02e8fed38159bb70f7e100e00f471eae2bc838cd0f4ebb21e28f8541 \
+    --hash=sha256:8c1be557ee92a20f184922c7b6424e8ab6691788e6d86137c5d93c1a6ec1b8fb \
+    --hash=sha256:bb4191dfc9306777bc594117aee052446b3fa88737cd13b7188d0e7aa8162185 \
+    --hash=sha256:bfb51918d4ff3d77c1c856a9699f8492c612cde32fd3bcd344af9be34999bfdc \
+    --hash=sha256:c20cfa2d49991c8b4147af39859b167664f2ad4561704ee74c1de03318e898db \
+    --hash=sha256:cb333c16912324fd5f769fff6bc5de372e9e7a202247b48870bc251ed40239aa \
+    --hash=sha256:d2d9808ea7b4af864f35ea216be506ecec180628aced0704e34aca0b040ffe46 \
+    --hash=sha256:d483ad4e639292c90170eb6f7783ad19490e7a8defb3e46f97dfe4bacae89122 \
+    --hash=sha256:dd5de0646207f053eb0d6c74ae45ba98c3395a571a2891858e87df7c9b9bd51b \
+    --hash=sha256:e1d4970ea66be07ae37a3c2e48b5ec63f7ba6804bdddfdbd3cfd954d25a82e63 \
+    --hash=sha256:e4fac90784481d221a8e4b1162afa7c47ed953be40d31ab4629ae917510051df \
+    --hash=sha256:fa5ae20527d8e831e8230cbffd9f8fe952815b2b7dae6ffec25318803a7528fc \
+    --hash=sha256:fd7f6999a8070df521b6384004ef42833b9bd62cfee11a09bda1079b4b704247 \
+    --hash=sha256:fdc842473cd33f45ff6bce46aea678a54e3d21f1b61a7750ce3c498eedfe25d6 \
+    --hash=sha256:fe69978f3f768926cfa37b867e3843918e012cf83f680806599ddce33c2c68b0
+    # via pre-commit
+six==1.16.0 \
+    --hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 \
+    --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
+    # via virtualenv
 tabulate==0.7.5 \
     --hash=sha256:9071aacbd97a9a915096c1aaf0dc684ac2672904cd876db5904085d6dac9810e
     # via -r requirements-dev.txt
 toml==0.10.2 \
     --hash=sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b \
     --hash=sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f
     # via
+    #   pre-commit
     #   pytest
     #   pytest-cov
 typing-extensions==3.10.0.2 \
     --hash=sha256:49f75d16ff11f1cd258e1b988ccff82a3ca5570217d7ad8c5f48205dd99a677e \
     --hash=sha256:d8226d10bc02a29bcc81df19a26e56a9647f8b0a6d4a83924139f4a8b01f17b7 \
     --hash=sha256:f1d25edafde516b146ecd0613dabcc61409817af4766fbbcfb8d1ad4ec441a34
     # via importlib-metadata
+virtualenv==20.8.1 \
+    --hash=sha256:10062e34c204b5e4ec5f62e6ef2473f8ba76513a9a617e873f1f8fb4a519d300 \
+    --hash=sha256:bcc17f0b3a29670dd777d6f0755a4c04f28815395bca279cdcb213b97199a6b8
+    # via pre-commit
 wheel==0.37.0 \
     --hash=sha256:21014b2bd93c6d0034b6ba5d35e4eb284340e09d63c59aef6fc14b0f346146fd \
     --hash=sha256:e2ef7239991699e3355d54f8e968a21bb940a1dbf34a4d226741e64462516fad
     # via -r requirements-dev.txt
 zipp==3.5.0 \
     --hash=sha256:957cfda87797e389580cb8b9e3870841ca991e2125350677b2ca83a0e99390a3 \
     --hash=sha256:f5812b1e007e48cff63449a5e9f4e7ebea716b4111f9c4f9a645f91d579bf0c4
-    # via importlib-metadata
+    # via
+    #   importlib-metadata
+    #   importlib-resources

requirements-dev.txt

@@ -3,6 +3,7 @@ tabulate==0.7.5
 nose==1.3.7
 coverage==5.5
 wheel==0.37.0
+pre-commit>=2.3.0
 
 # Pytest specific deps
 pytest==6.2.5

s3transfer/bandwidth.py

@@ -274,7 +274,7 @@ def _release_requested_amt_for_scheduled_request(self, amt, request_token,
         return self._release_requested_amt(amt, time_now)
 
     def _raise_request_exceeded_exception(self, amt, request_token, time_now):
-        allocated_time = amt/float(self._max_rate)
+        allocated_time = amt / float(self._max_rate)
         retry_time = self._consumption_scheduler.schedule_consumption(
             amt, request_token, allocated_time)
         raise RequestExceededException(

s3transfer/compat.py

@@ -98,4 +98,5 @@ def fallocate(fileobj, size):
         fileobj.truncate(size)
 
 
-from multiprocessing.managers import BaseManager
+# Import at end of file to avoid circular dependencies
+from multiprocessing.managers import BaseManager  # noqa: F401,E402

s3transfer/copies.py

@@ -193,7 +193,8 @@ def _submit_multipart_request(self, client, config, osutil,
             # range parameter starts at zero, so just subtract 1 off of
             # the part number
             extra_part_args['CopySourceRange'] = calculate_range_parameter(
-                part_size, part_number-1, num_parts, transfer_future.meta.size)
+                part_size, part_number-1, num_parts, transfer_future.meta.size
+            )
             # Get the size of the part copy as well for the progress
             # callbacks.
             size = self._get_transfer_size(

s3transfer/download.py

@@ -259,8 +259,10 @@ def __init__(self, osutil, transfer_coordinator, io_executor,
 
     @classmethod
     def is_compatible(cls, download_target, osutil):
-        return isinstance(download_target, six.string_types) and \
-               osutil.is_special_file(download_target)
+        return (
+            isinstance(download_target, six.string_types)
+            and osutil.is_special_file(download_target)
+        )
 
     def get_fileobj_for_io_writes(self, transfer_future):
         filename = transfer_future.meta.call_args.fileobj

s3transfer/processpool.py

@@ -353,10 +353,10 @@ def download_file(self, bucket, key, filename, extra_args=None,
         self._validate_all_known_args(extra_args)
         transfer_id = self._transfer_monitor.notify_new_transfer()
         download_file_request = DownloadFileRequest(
-                transfer_id=transfer_id, bucket=bucket, key=key,
-                filename=filename, extra_args=extra_args,
-                expected_size=expected_size,
-            )
+            transfer_id=transfer_id, bucket=bucket, key=key,
+            filename=filename, extra_args=extra_args,
+            expected_size=expected_size,
+        )
         logger.debug(
             'Submitting download file request: %s.', download_file_request)
         self._download_request_queue.put(download_file_request)
@@ -927,9 +927,10 @@ def _do_get_object(self, bucket, key, extra_args, temp_filename, offset):
                 self._write_to_file(temp_filename, offset, response['Body'])
                 return
             except S3_RETRYABLE_DOWNLOAD_ERRORS as e:
-                logger.debug('Retrying exception caught (%s), '
-                             'retrying request, (attempt %s / %s)', e, i+1,
-                             self._MAX_ATTEMPTS, exc_info=True)
+                logger.debug(
+                    'Retrying exception caught (%s), '
+                    'retrying request, (attempt %s / %s)', e, i+1,
+                    self._MAX_ATTEMPTS, exc_info=True)
                 last_exception = e
         raise RetriesExceededError(last_exception)
 

s3transfer/upload.py

@@ -146,7 +146,7 @@ def stores_body_in_memory(self, operation_name):
             memory. False if the manager will not directly store the body in
             memory.
         """
-        raise NotImplemented('must implement store_body_in_memory()')
+        raise NotImplementedError('must implement store_body_in_memory()')
 
     def provide_transfer_size(self, transfer_future):
         """Provides the transfer size of an upload

s3transfer/utils.py

@@ -11,7 +11,6 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 import random
-import time
 import functools
 import math
 import os
@@ -20,15 +19,13 @@
 import string
 import logging
 import threading
-import io
 from collections import defaultdict
 
 from botocore.exceptions import IncompleteReadError
 from botocore.exceptions import ReadTimeoutError
 
 from s3transfer.compat import SOCKET_ERROR
 from s3transfer.compat import rename_file
-from s3transfer.compat import seekable
 from s3transfer.compat import fallocate
 
 
@@ -305,7 +302,7 @@ def get_temp_filename(self, filename):
         suffix = os.extsep + random_file_extension()
         path = os.path.dirname(filename)
         name = os.path.basename(filename)
-        temp_filename = name[:self._MAX_FILENAME_LEN - len(suffix)] + suffix 
+        temp_filename = name[:self._MAX_FILENAME_LEN - len(suffix)] + suffix
         return os.path.join(path, temp_filename)
 
     def allocate(self, filename, size):

scripts/ci/run-crt-tests

@@ -29,12 +29,12 @@ def run(command, env=None):
 
 
 try:
-    import awscrt
+    import awscrt  # noqa: F401
 except ImportError:
     print("MISSING DEPENDENCY: awscrt must be installed to run the crt tests.")
     sys.exit(1)
 
 
 if __name__ == "__main__":
     with cd(os.path.join(REPO_ROOT, "tests")):
-       run(f"{REPO_ROOT}/scripts/ci/run-tests unit/ functional/")
+        run(f"{REPO_ROOT}/scripts/ci/run-tests unit/ functional/")