Skip to content

Commit

Permalink
🧵 Safer downloads (#673)
Browse files Browse the repository at this point in the history
Files are first downloaded as temporary files and then transferred to the targeted location. This prevents corrupted files from being stored if the download is terminated.

------

Co-authored-by: Shan E Ahmed Raza <13048456+shaneahmed@users.noreply.github.com>
  • Loading branch information
blaginin and shaneahmed authored Aug 14, 2023
1 parent 8615181 commit a157833
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions tiatoolbox/utils/misc.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

import copy
import json
import shutil
import tempfile
import zipfile
from pathlib import Path
from typing import IO, TYPE_CHECKING
Expand Down Expand Up @@ -714,9 +716,13 @@ def download_data(
# Raise an exception for status codes != 200
response.raise_for_status()
# Write the file in blocks of 1024 bytes to avoid running out of memory
with save_path.open("wb") as handle:

with tempfile.NamedTemporaryFile(mode="wb", delete=False) as templ_file:
for block in response.iter_content(1024):
handle.write(block)
templ_file.write(block)

# Move the temporary file to the desired location
shutil.move(templ_file.name, save_path)

if unzip:
unzip_path = save_dir / save_path.stem
Expand Down

0 comments on commit a157833

Please sign in to comment.