Catch get_bucket_location IOError in send_file #502

karlvr · 2015-03-17T05:16:37Z

I frequently see a broken pipe, which I tracked down to get_bucket_location. This seems to happen on big syncs, where there are lots of files to compare - maybe the connection is being dropped?

Borrowing the retry code from the try block below, this simply retries the same send_file, which appears to solve the problem.

(Confession: this is the first Python I've edited / written so all feedback received gladly and I'll happily tidy up / modify, hand-over or rewrite this)

I frequently see a broken pipe, which I tracked down to get_bucket_location. This seems to happen on big syncs, where there are lots of files to compare - maybe the connection is being dropped? Borrowing the retry code from the try block below, this simply retries the same send_file, which appears to solve the problem.

mdomsch · 2015-03-17T19:19:35Z

this isn't quite right, but if you can frequently get a broken pipe message, please try with https://github.com/mdomsch/s3cmd/tree/bug/epipe-2 branch and see if that addresses it correctly. The retry logic needs to be done at a lower level than where you put it in your patch.

karlvr · 2015-03-18T01:24:37Z

@mdomsch Thank you, I'll try that out and report back tomorrow.

karlvr · 2015-03-20T20:57:30Z

I didn't update to that branch the other day - my patch did appear to address the issue that I was seeing. I'm now running with bug/epipe-2 from your repo (which appeared to be simply master?) and will report back as previously promised.

karlvr · 2015-03-21T19:31:52Z

@mdomsch Running with the bug/epipe-2 branch, I received the following error overnight. This appears to be an IOError in the get_bucket_location. Am I definitely using the right branch for testing?

INFO: Summary: 1 local files to upload, 0 files to remote copy, 0 remote files to delete

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   An unexpected error has occurred.
 Please try reproducing the error using
 the latest s3cmd code from the git master
 branch found at:
   https://github.com/s3tools/s3cmd
 If the error persists, please report the
 following lines (removing any private
 info as necessary) to:
  s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/bin/s3cmd --config=/tmp/s3cmd.cfg.vndUa3Z6Ss --human-readable-sizes --delete-removed --exclude share/* --exclude Temporary* --exclude tmp --exclude Trash --rexclude work/[^/]+/assets.* --exclude */.svn -v sync /srv/cactuslab/ s3://cactuslab-backup-dev1/cactuslab/
Problem: error: [Errno 32] Broken pipe
S3cmd:   1.5.2
python:   2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2]
environment LANG=en_NZ.UTF-8

Traceback (most recent call last):
 File "/usr/bin/s3cmd", line 2556, in <module>
   rc = main()
 File "/usr/bin/s3cmd", line 2468, in main
   rc = cmd_func(args)
 File "/usr/bin/s3cmd", line 1486, in cmd_sync
   return cmd_sync_local2remote(args)
 File "/usr/bin/s3cmd", line 1463, in cmd_sync_local2remote
   any_child_failed = _single_process(local_list)
 File "/usr/bin/s3cmd", line 1268, in _single_process
   rc = _child(destination_base, local_list)
 File "/usr/bin/s3cmd", line 1405, in _child
   n, total_size = _upload(update_list, n, upload_count, total_size)
 File "/usr/bin/s3cmd", line 1328, in _upload
   response = s3.object_put(src, uri, extra_headers, extra_label = seq_label)
 File "/usr/share/s3cmd/S3/S3.py", line 595, in object_put
   response = self.send_file(request, file, labels)
 File "/usr/share/s3cmd/S3/S3.py", line 1035, in send_file
   region = self.get_bucket_location(s3_uri)
 File "/usr/share/s3cmd/S3/S3.py", line 345, in get_bucket_location
   response = self.send_request(request)
 File "/usr/share/s3cmd/S3/S3.py", line 970, in send_request
   conn.c.request(method_string, uri, request.body, headers)
 File "/usr/lib/python2.7/httplib.py", line 973, in request
   self._send_request(method, url, body, headers)
 File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
   self.endheaders(body)
 File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
   self._send_output(message_body)
 File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
   self.send(msg)
 File "/usr/lib/python2.7/httplib.py", line 805, in send
   self.sock.sendall(data)
 File "/usr/lib/python2.7/ssl.py", line 329, in sendall
   v = self.send(data[count:])
 File "/usr/lib/python2.7/ssl.py", line 298, in send
   v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   An unexpected error has occurred.
 Please try reproducing the error using
 the latest s3cmd code from the git master
 branch found at:
   https://github.com/s3tools/s3cmd
 If the error persists, please report the
 above lines (removing any private
 info as necessary) to:
  s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*** WARN: Failed with status 74

mdomsch · 2015-03-24T20:38:27Z

Please pull the bug/epipe-2 branch again and retry. I added more cases of
catching the EPIPE.

On Sat, Mar 21, 2015 at 2:31 PM, Karl von Randow notifications@github.com
wrote:

@mdomsch https://github.com/mdomsch Running with the bug/epipe-2
branch, I received the following error overnight. This appears to be an
IOError in the get_bucket_location. Am I definitely using the right branch
for testing?

INFO: Summary: 1 local files to upload, 0 files to remote copy, 0 remote files to delete

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/bin/s3cmd --config=/tmp/s3cmd.cfg.vndUa3Z6Ss --human-readable-sizes --delete-removed --exclude share/* --exclude Temporary* --exclude tmp --exclude Trash --rexclude work/[^/]+/assets.* --exclude */.svn -v sync /srv/cactuslab/ s3://cactuslab-backup-dev1/cactuslab/
Problem: error: [Errno 32] Broken pipe
S3cmd: 1.5.2
python: 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2]
environment LANG=en_NZ.UTF-8

Traceback (most recent call last):
File "/usr/bin/s3cmd", line 2556, in
rc = main()
File "/usr/bin/s3cmd", line 2468, in main
rc = cmd_func(args)
File "/usr/bin/s3cmd", line 1486, in cmd_sync
return cmd_sync_local2remote(args)
File "/usr/bin/s3cmd", line 1463, in cmd_sync_local2remote
any_child_failed = _single_process(local_list)
File "/usr/bin/s3cmd", line 1268, in _single_process
rc = _child(destination_base, local_list)
File "/usr/bin/s3cmd", line 1405, in _child
n, total_size = _upload(update_list, n, upload_count, total_size)
File "/usr/bin/s3cmd", line 1328, in _upload
response = s3.object_put(src, uri, extra_headers, extra_label = seq_label)
File "/usr/share/s3cmd/S3/S3.py", line 595, in object_put
response = self.send_file(request, file, labels)
File "/usr/share/s3cmd/S3/S3.py", line 1035, in send_file
region = self.get_bucket_location(s3_uri)
File "/usr/share/s3cmd/S3/S3.py", line 345, in get_bucket_location
response = self.send_request(request)
File "/usr/share/s3cmd/S3/S3.py", line 970, in send_request
conn.c.request(method_string, uri, request.body, headers)
File "/usr/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 805, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/ssl.py", line 329, in sendall
v = self.send(data[count:])
File "/usr/lib/python2.7/ssl.py", line 298, in send
v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*** WARN: Failed with status 74

—
Reply to this email directly or view it on GitHub
#502 (comment).

karlvr · 2015-03-25T04:15:37Z

Thanks @mdomsch, I've installed it and I'll let you know the results!

mdomsch · 2015-03-27T03:42:41Z

bug/epipe-2 merged to master now. Please report any additional failures that result. Possibly one in the CloudFront path but as that needs a lot of work in general, I didn't touch it for now.

mdomsch closed this Mar 27, 2015

fviard mentioned this pull request Aug 25, 2016

Meta-ticket - Add support for Expect: 100-continue #778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Catch get_bucket_location IOError in send_file #502

Catch get_bucket_location IOError in send_file #502

karlvr commented Mar 17, 2015

mdomsch commented Mar 17, 2015

karlvr commented Mar 18, 2015

karlvr commented Mar 20, 2015

karlvr commented Mar 21, 2015

mdomsch commented Mar 24, 2015

karlvr commented Mar 25, 2015

mdomsch commented Mar 27, 2015

Catch get_bucket_location IOError in send_file #502

Catch get_bucket_location IOError in send_file #502

Conversation

karlvr commented Mar 17, 2015

mdomsch commented Mar 17, 2015

karlvr commented Mar 18, 2015

karlvr commented Mar 20, 2015

karlvr commented Mar 21, 2015

mdomsch commented Mar 24, 2015

karlvr commented Mar 25, 2015

mdomsch commented Mar 27, 2015