Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

s3cmd: put fails with “Connection reset by peer” for large files

DZone's Guide to

s3cmd: put fails with “Connection reset by peer” for large files

· DevOps Zone
Free Resource

Download the blueprint that can take a company of any maturity level all the way up to enterprise-scale continuous delivery using a combination of Automic Release Automation, Automic’s 20+ years of business automation experience, and the proven tools and practices the company is already leveraging.

I recently wanted to copy some large files from an AWS instance into an S3 bucket using s3cmd but ended up with the following error when trying to use the ‘put’ command:

$ s3cmd put /mnt/ebs/myfile.tar s3://mybucket.somewhere.com
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [1 of 1]
     1077248 of 12185313280     0% in    1s   937.09 kB/s  failed
WARNING: Upload failed: /myfile.tar ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [1 of 1]
     1183744 of 12185313280     0% in    1s  1062.18 kB/s  failed
WARNING: Upload failed: /myfile.tar ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [1 of 1]
      417792 of 12185313280     0% in    1s   378.75 kB/s  failed
WARNING: Upload failed: /myfile.tar ([Errno 104] Connection reset by peer)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [1 of 1]
       94208 of 12185313280     0% in    1s    81.04 kB/s  failed
WARNING: Upload failed: /myfile.tar ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [1 of 1]
       28672 of 12185313280     0% in    1s    18.40 kB/s  failed
WARNING: Upload failed: /myfile.tar ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [1 of 1]
       12288 of 12185313280     0% in    2s     4.41 kB/s  failed
ERROR: Upload of '/mnt/ebs/myfile.tar' failed too many times. Skipping that file.

I tried with a smaller file just to make sure I wasn’t doing anything stupid syntax wise and that transferred without a problem which lead me to believe the problem might be when uploading larger files – the one I was uploading was around ~10GB in size.

I eventually came across this StackOverflow thread which suggested that files >5GB in size need to make use of the ‘multi part method’ which was released in version 1.1.0 of s3cmd.

The Ubuntu repository comes with version 1.0.0 so I needed to find a way of getting a newer version onto the machine.

I eventually ended up downloading version 1.5.0 from sourceforge but I couldn’t get a direct URI to download it so I ended up downloading it to my machine, uploading to the S3 bucket through the web UI and then pulling it back down again using a ‘s3cmd get’. #epic

In retrospect the s3cmd PPA might have been a better option.

Anyway, when I used this s3cmd it uploaded using multi part fine:

...
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [part 761 of 775, 15MB]
 15728640 of 15728640   100% in    3s     4.12 MB/s  done
/mnt/ebs/myfile.tar -> s3://mybucket.somewhere.com/myfile.tar  [part 762 of 775, 15MB]
...

Download the ‘Practical Blueprint to Continuous Delivery’ to learn how Automic Release Automation can help you begin or continue your company’s digital transformation.

Topics:

Published at DZone with permission of Mark Needham, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}