Regression: Multi chunk transfer hangs as merging chunks fails

[clehene]

Description

This is caused by 4b3171d which removed _submit to achieve better control of done callback behavior and avoid a race condition.

However _submit was called when merging chunks and now fails.

The behavior is that the transfer will hang after the error. I'm not clear why it hangs instead of actually bubbling the error up (that may be due to a separate issue).

I'll submit a patch once I finish testing. Note that this affects latest release of the CLI.

azure.datalake.store.transfer : Chunks transferred
azure.datalake.store.transfer : Merging file: transferring
concurrent.futures : exception calling callback for <Future at 0x10faa1610 state=finished returned tuple>
Traceback (most recent call last):
  File "/Users/clehene/work/azure-cli/env/lib/python2.7/site-packages/concurrent/futures/_base.py", line 301, in _invoke_callbacks
    callback(self)
  File "/Users/clehene/work/azure-cli/env/lib/python2.7/site-packages/azure/datalake/store/transfer.py", line 434, in _update
    merge_future = self._submit(
AttributeError: 'ADLTransferClient' object has no attribute '_submit'

Interrupting the process after the exception will "wake it":

^Cazure.datalake.store.transfer : <azure.datalake.store.transfer.ADLTransferClient object at 0x104fba290> suspended and persisted
azure.datalake.store.transfer : Shutting down worker threads
azure.datalake.store.transfer : Shutdown complete
Application event 'Application.TransformResults' with event data {'event_data': {'result': None}}
Application event 'Application.FilterResults' with event data {'event_data': {'result': None}}

---

Reproduction Steps

• Upload a file and make sure the chunk size is smaller than the file size

Environment summary

SDK Version: What version of the SDK are you using? (pip show azure-datalake-store)

Name: azure-datalake-store
Version: 0.0.15

[clehene]

Merged