Skip to content

TypeError in S3Client.copy() when threads=1 #3355

@ttcm3010

Description

@ttcm3010

Description

​ The test test_min_threads_enforcement fails with a TypeError because _copy_file.call_args returns None, implying _copy_file() was never called during the execution of S3Client.copy() when threads=1.

Steps to Reproduce

Minimal reproducible test:

import os
import sys
import tempfile
import unittest
from datetime import datetime, timedelta
from unittest.mock import MagicMock, patch
from luigi.contrib.s3 import S3Client
from luigi.target import MissingParentDirectory

class TestS3Copy(unittest.TestCase):

    def setUp(self):
        self.client = S3Client(aws_access_key_id='fake_key',
            aws_secret_access_key='fake_secret')
        self.client._s3 = MagicMock()
        self.client.isdir = MagicMock()
        self.client._copy_dir = MagicMock(return_value=(3, 1024))
        self.client._copy_file = MagicMock(return_value=(1, 512))
        
	def test_min_threads_enforcement(self):
    	"""Test minimum thread count enforcement"""
    	self.client.copy('s3://src/file', 's3://dest/file', threads=1)
    	args, kwargs = self.client._copy_file.call_args

This test assumes _copy_file() will be called when copying a single file with threads=1.

Expected behavior:

When copying a single S3 file with threads=1, the copy() method should internally call _copy_file(), and the test should be able to inspect its call arguments.

Actual behavior:

The copy() method does not invoke _copy_file(), resulting in call_args being None. Attempting to unpack None raises a TypeError.

the traceback is:

___________________ TestS3Copy.test_min_threads_enforcement ____________________

self = <contrib.test_copy_tttmp.TestS3Copy testMethod=test_min_threads_enforcement>

    def test_min_threads_enforcement(self):
        """Test minimum thread count enforcement"""
        self.client.copy('s3://src/file', 's3://dest/file', threads=1)
>       args, kwargs = self.client._copy_file.call_args
E       TypeError: cannot unpack non-iterable NoneType object

test/contrib/test_copy_tttmp.py:40: TypeError

Reproduces How Often: Always

Versions:

  • Luigi: master

  • Python: 3.10.16

  • pytest: 8.3.5

  • Platform: Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions