Sourcery refactored main branch - #1
Open
sourcery-ai[bot] wants to merge 1 commit into
Open
Conversation
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
| Bytes : lz4 | ||
|
|
||
| """ | ||
|
|
Author
There was a problem hiding this comment.
Lines 81-83 refactored with the following changes:
- Convert for loop into dictionary comprehension (
dict-comprehension)
| ENCODED_CHUNK_NAMES_FOLDER = "chunks_index" | ||
| # unsharded naming will help with backwards compatibility | ||
| ENCODED_CHUNK_NAMES_FILENAME = f"unsharded" | ||
| ENCODED_CHUNK_NAMES_FILENAME = 'unsharded' |
Author
There was a problem hiding this comment.
Lines 56-56 refactored with the following changes:
- Replace f-string with no interpolated values with string (
remove-redundant-fstring)
Comment on lines
+263
to
-269
| elif isinstance(e, (DatasetHandlerError, PathNotEmptyException)): | ||
| raise DatasetHandlerError( | ||
| "A Hub dataset wasn't found at the specified path. " | ||
| "This may be due to a corrupt dataset or a wrong path. " | ||
| "If you want to delete the data at the path regardless, use force=True" | ||
| ) | ||
| else: | ||
| if isinstance(e, (DatasetHandlerError, PathNotEmptyException)): | ||
| raise DatasetHandlerError( | ||
| "A Hub dataset wasn't found at the specified path. " | ||
| "This may be due to a corrupt dataset or a wrong path. " | ||
| "If you want to delete the data at the path regardless, use force=True" | ||
| ) |
Author
There was a problem hiding this comment.
Function dataset.delete refactored with the following changes:
- Lift code into else after jump in control flow (
reintroduce-else) - Merge else clause's nested if statement into elif (
merge-else-if-into-elif)
Comment on lines
-478
to
+483
| if os.path.isdir(src) and os.path.isdir(dest): | ||
| if os.path.samefile(src, dest): | ||
| raise SamePathException(src) | ||
| if ( | ||
| os.path.isdir(src) | ||
| and os.path.isdir(dest) | ||
| and os.path.samefile(src, dest) | ||
| ): | ||
| raise SamePathException(src) |
Author
There was a problem hiding this comment.
Function dataset.ingest_kaggle refactored with the following changes:
- Merge nested if conditions (
merge-nested-ifs) - Inline variable that is immediately returned (
inline-immediately-returned-variable)
Comment on lines
-519
to
+520
| datasets = client.get_datasets(workspace=workspace) | ||
| return datasets | ||
| return client.get_datasets(workspace=workspace) |
Author
There was a problem hiding this comment.
Function dataset.list refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
| self.token = self.get_token() | ||
| else: | ||
| self.token = token | ||
| self.token = self.get_token() if token is None else token |
Author
There was a problem hiding this comment.
Function HubBackendClient.__init__ refactored with the following changes:
- Replace if statement with if expression (
assign-if-exp)
Comment on lines
-229
to
-245
| response = self.request( | ||
| return self.request( | ||
| "GET", | ||
| suffix_user, | ||
| endpoint=self.endpoint(), | ||
| params={"organization": workspace}, | ||
| ).json() | ||
| else: | ||
| print( | ||
| f'You are not a member of organization "{workspace}". List of accessible datasets from "{workspace}": ', | ||
| ) | ||
| response = self.request( | ||
| print( | ||
| f'You are not a member of organization "{workspace}". List of accessible datasets from "{workspace}": ', | ||
| ) | ||
| return self.request( | ||
| "GET", | ||
| suffix_public, | ||
| endpoint=self.endpoint(), | ||
| params={"organization": workspace}, | ||
| ).json() | ||
| return response |
Author
There was a problem hiding this comment.
Function HubBackendClient.get_workspace_datasets refactored with the following changes:
- Lift return into if (
lift-return-into-if) - Remove unnecessary else after guard condition (
remove-unnecessary-else)
Comment on lines
-496
to
+497
| if not aslist and last_shape is not None: | ||
| if shape != last_shape: | ||
| raise DynamicTensorNumpyError(self.key, index, "shape") | ||
| if not aslist and last_shape is not None and shape != last_shape: | ||
| raise DynamicTensorNumpyError(self.key, index, "shape") |
Author
There was a problem hiding this comment.
Function ChunkEngine.numpy refactored with the following changes:
- Merge nested if conditions (
merge-nested-ifs)
Comment on lines
-197
to
+203
| if compression == "lz4": | ||
| if ( | ||
| buffer[:4] == b'\x04"M\x18' | ||
| ): # python-lz4 magic number (backward compatiblity) | ||
| return lz4.frame.decompress(buffer) | ||
| return numcodecs.lz4.decompress(buffer) | ||
| else: | ||
| if compression != "lz4": | ||
| raise SampleDecompressionError() | ||
| if ( | ||
| buffer[:4] == b'\x04"M\x18' | ||
| ): # python-lz4 magic number (backward compatiblity) | ||
| return lz4.frame.decompress(buffer) | ||
| return numcodecs.lz4.decompress(buffer) |
Author
There was a problem hiding this comment.
Function decompress_bytes refactored with the following changes:
- Swap if/else branches (
swap-if-else-branches) - Remove unnecessary else after guard condition (
remove-unnecessary-else)
Comment on lines
-419
to
+418
| elif compression in ("mp4", "mkv", "avi"): | ||
| elif compression in {"mp4", "mkv", "avi"}: |
Author
There was a problem hiding this comment.
Function verify_compressed_file refactored with the following changes:
- Use set when checking membership of a collection of literals (
collection-into-set)
Comment on lines
-555
to
+554
| if img.mode == 1: | ||
| args = ("L",) | ||
| else: | ||
| args = (img.mode,) | ||
| args = ("L", ) if img.mode == 1 else (img.mode, ) |
Author
There was a problem hiding this comment.
Function _fast_decompress refactored with the following changes:
- Replace if statement with if expression (
assign-if-exp)
Comment on lines
-723
to
+719
| if bits == 8: | ||
| nlayers = 2 | ||
| else: | ||
| nlayers = 4 | ||
| nlayers = 2 if bits == 8 else 4 |
Author
There was a problem hiding this comment.
Function _read_png_shape_and_dtype refactored with the following changes:
- Replace if statement with if expression (
assign-if-exp)
Comment on lines
-848
to
+841
| if "duration" in ret: | ||
| ret["duration"] = float(ret["duration"]) | ||
| else: | ||
| ret["duration"] = duration | ||
| ret["duration"] = float(ret["duration"]) if "duration" in ret else duration |
Author
There was a problem hiding this comment.
Function _get_video_info refactored with the following changes:
- Replace if statement with if expression (
assign-if-exp)
Comment on lines
-860
to
+850
| return sum([60 ** i * float(j) for (i, j) in enumerate(time.split(":")[::-1])]) | ||
| return sum(60 ** i * float(j) for (i, j) in enumerate(time.split(":")[::-1])) |
Author
There was a problem hiding this comment.
Function to_seconds refactored with the following changes:
- Replace unneeded comprehension with generator (
comprehension-to-generator)
Comment on lines
-23
to
+26
| for i in range(m, n): | ||
| for _ in range(m, n): | ||
| arr2.append(0) | ||
| elif m > n: | ||
| for i in range(n, m): | ||
| for _ in range(n, m): |
Author
There was a problem hiding this comment.
Function version_compare refactored with the following changes:
- Replace unused for index with underscore (
for-index-underscore)
| return IndexEntry(new_value) | ||
| elif isinstance(self.value, tuple): | ||
| if isinstance(item, int) or isinstance(item, slice): | ||
| if isinstance(item, (int, slice)): |
Author
There was a problem hiding this comment.
Function IndexEntry.__getitem__ refactored with the following changes:
- Merge isinstance calls (
merge-isinstance)
Comment on lines
-192
to
+193
| and self.value.stop == None | ||
| and ((self.value.step or 1) == 1) | ||
| and self.value.stop is None | ||
| and (self.value.step or 1) == 1 |
Author
There was a problem hiding this comment.
Function IndexEntry.is_trivial refactored with the following changes:
- Use x is None rather than x == None (
none-compare)
Comment on lines
-237
to
+242
| if isinstance(self.value, int): | ||
| if self.value >= parent_length or self.value < -parent_length: | ||
| raise ValueError( | ||
| f"Index {self.value} is out of range for tensors with length {parent_length}" | ||
| ) | ||
| if isinstance(self.value, int) and ( | ||
| self.value >= parent_length or self.value < -parent_length | ||
| ): | ||
| raise ValueError( | ||
| f"Index {self.value} is out of range for tensors with length {parent_length}" | ||
| ) |
Author
There was a problem hiding this comment.
Function IndexEntry.validate refactored with the following changes:
- Merge nested if conditions (
merge-nested-ifs)
Comment on lines
-301
to
+303
| else: | ||
| new_values = self.values[:i] + [self.values[i][item]] + self.values[i + 1 :] | ||
| return Index(new_values) | ||
| new_values = self.values[:i] + [self.values[i][item]] + self.values[i + 1 :] | ||
| return Index(new_values) |
Author
There was a problem hiding this comment.
Function Index.compose_at refactored with the following changes:
- Remove unnecessary else after guard condition (
remove-unnecessary-else)
| or compatible with NumPy's advanced integer indexing. | ||
| """ | ||
| if isinstance(item, int) or isinstance(item, slice): | ||
| if isinstance(item, (int, slice)): |
Author
There was a problem hiding this comment.
Function Index.__getitem__ refactored with the following changes:
- Merge isinstance calls (
merge-isinstance)
| """ | ||
| index_values = tuple(item.value for item in self.values[1:]) | ||
| samples = list(arr[index_values] for arr in samples) | ||
| samples = [arr[index_values] for arr in samples] |
Author
There was a problem hiding this comment.
Function Index.apply refactored with the following changes:
- Replace list(), dict() or set() with comprehension (
collection-builtin-to-comprehension)
Comment on lines
-136
to
-144
| required_meta = { | ||
| return { | ||
| "htype": htype, | ||
| "min_shape": [], | ||
| "max_shape": [], | ||
| "length": 0, | ||
| **defaults, | ||
| } | ||
|
|
||
| return required_meta |
Author
There was a problem hiding this comment.
Function _required_meta_from_htype refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
Comment on lines
-156
to
+158
| if value == UNSPECIFIED: | ||
| if defaults[key] == REQUIRE_USER_SPECIFICATION: | ||
| raise TensorMetaMissingRequiredValue(htype, key) | ||
| if ( | ||
| value == UNSPECIFIED | ||
| and defaults[key] == REQUIRE_USER_SPECIFICATION | ||
| ): | ||
| raise TensorMetaMissingRequiredValue(htype, key) |
Author
There was a problem hiding this comment.
Function _validate_htype_overwrites refactored with the following changes:
- Merge nested if conditions (
merge-nested-ifs) - Use set when checking membership of a collection of literals (
collection-into-set)
Comment on lines
-216
to
+218
| if htype in ("json", "list", "text") and not htype_overwrite["dtype"]: | ||
| if htype in {"json", "list", "text"} and not htype_overwrite["dtype"]: |
Author
There was a problem hiding this comment.
Function _replace_unspecified_values refactored with the following changes:
- Use set when checking membership of a collection of literals (
collection-into-set)
Comment on lines
-239
to
+241
| if htype in ("json", "list"): | ||
| if htype in {"json", "list"}: |
Author
There was a problem hiding this comment.
Function _validate_required_htype_overwrites refactored with the following changes:
- Merge nested if conditions (
merge-nested-ifs) - Swap positions of nested conditionals (
swap-nested-ifs) - Hoist repeated code outside conditional statement (
hoist-statement-from-if) - Use set when checking membership of a collection of literals (
collection-into-set)
Comment on lines
-138
to
+143
| dsv = local_ds[0:3] | ||
| dsv = local_ds[:3] | ||
| dsv.abc[0] = 5 | ||
| h = local_ds.commit("it is 5") | ||
| i = local_ds.checkout(e) | ||
| local_ds.abc[0] = 6 | ||
| tsv = local_ds.abc[0:5] | ||
| tsv = local_ds.abc[:5] |
Author
There was a problem hiding this comment.
Function test_auto_checkout_bug refactored with the following changes:
- Replace [0:x] with [:x] (
remove-redundant-slice-index)
Comment on lines
+36
to
+39
| if ensure_order: | ||
| return MultiThreadedNaiveScheduler(num_workers) | ||
| else: | ||
| if ensure_order: | ||
| return MultiThreadedNaiveScheduler(num_workers) | ||
| else: | ||
| return SequentialMultithreadScheduler(num_workers) | ||
| return SequentialMultithreadScheduler(num_workers) |
Author
There was a problem hiding this comment.
Function use_scheduler refactored with the following changes:
- Remove unnecessary else after guard condition (
remove-unnecessary-else)
Comment on lines
-293
to
+317
| if not self.shutdown: | ||
| self.workers_done.set() | ||
| if self.shutdown: | ||
| return | ||
|
|
||
| try: | ||
| for wid in range(self.num_workers): | ||
| self.request_queues[wid].put(None) | ||
| self.workers[wid].join(timeout=5) | ||
|
|
||
| for queue in self.request_queues: | ||
| queue.cancel_join_thread() | ||
| queue.close() | ||
|
|
||
| self.data_queue.cancel_join_thread() | ||
| self.data_queue.close() | ||
| finally: | ||
| if self._worker_pids_set: | ||
| _remove_worker_pids(id(self)) | ||
| self._worker_pids_set = False | ||
|
|
||
| for worker in self.workers: | ||
| if worker.is_alive(): | ||
| worker.terminate() | ||
|
|
||
| self.shutdown = True | ||
| self.workers_done.set() | ||
|
|
||
| try: | ||
| for wid in range(self.num_workers): | ||
| self.request_queues[wid].put(None) | ||
| self.workers[wid].join(timeout=5) | ||
|
|
||
| for queue in self.request_queues: | ||
| queue.cancel_join_thread() | ||
| queue.close() | ||
|
|
||
| self.data_queue.cancel_join_thread() | ||
| self.data_queue.close() | ||
| finally: | ||
| if self._worker_pids_set: | ||
| _remove_worker_pids(id(self)) | ||
| self._worker_pids_set = False | ||
|
|
||
| for worker in self.workers: | ||
| if worker.is_alive(): | ||
| worker.terminate() | ||
|
|
||
| self.shutdown = True |
Author
There was a problem hiding this comment.
Function PrefetchConcurrentIterator._shutdown_all refactored with the following changes:
- Add guard clause (
last-if-guard)
Comment on lines
-79
to
+83
| warning_message += f"Windows OS detected. Pytorch iteration speeds are up to 500% faster using linux/macOS along with Python version >= 3.8. " | ||
| warning_message += 'Windows OS detected. Pytorch iteration speeds are up to 500% faster using linux/macOS along with Python version >= 3.8. ' | ||
|
|
||
| else: | ||
| warning_message += f"Python version < 3.8 detected. Pytorch iteration speeds are up to 500% faster on Python version >= 3.8.\n" | ||
| warning_message += 'Python version < 3.8 detected. Pytorch iteration speeds are up to 500% faster on Python version >= 3.8.\n' | ||
|
|
Author
There was a problem hiding this comment.
Function TorchDataset.__init__ refactored with the following changes:
- Replace f-string with no interpolated values with string (
remove-redundant-fstring)
| buffer_len = len(self.buffer) | ||
|
|
||
| if not sample is None: | ||
| if sample is not None: |
Author
There was a problem hiding this comment.
Function ShuffleBuffer.exchange refactored with the following changes:
- Swap if/else branches (
swap-if-else-branches) - Remove unnecessary else after guard condition (
remove-unnecessary-else) - Hoist repeated code outside conditional statement (
hoist-statement-from-if) - Simplify logical expression using De Morgan identities (
de-morgan)
Author
Sourcery Code Quality Report✅ Merging this PR will increase code quality in the affected files by 0.21%.
Here are some functions in these files that still need a tune-up:
Legend and ExplanationThe emojis denote the absolute quality of the code:
The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request. Please see our documentation here for details on how these metrics are calculated. We are actively working on this report - lots more documentation and extra metrics to come! Help us improve this quality report! |
wjptak
approved these changes
Dec 22, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch
mainrefactored by Sourcery.If you're happy with these changes, merge this Pull Request using the Squash and merge strategy.
See our documentation here.
Run Sourcery locally
Reduce the feedback loop during development by using the Sourcery editor plugin:
Review changes via command line
To manually merge these changes, make sure you're on the
mainbranch, then run:Help us improve this pull request!