Resuming uploads to Amazon S3

Published on 2018/02/24

|

Last updated on 2018/03/27

|

by Nejc Zupan

Featured post image

I am escaping continental winter at the moment which means my internet connection is not perfect. I was uploading a 1GB file to Amazon S3 via the console.aws.amazon.com and the upload failed at 93%. Just my luck!

I didn’t want this to happen again. So I tried using Cyberduck to upload and intentionally turned off WiFi after a few seconds to see if I can resume the upload. Nope. Does not work. OK, what about the official AWS command-line tool? Also nope. Oh, come on!

Finally, I found this StackOverflow entry. Apparently `s3cmd` does support resuming uploads. Let’s give it a try:

~$ s3cmd put --multipart-chunk-size-mb=5 IRL_talk_recording.mp4 s3://videos.niteo.co/IRL4/
upload: 'IRL_talk_recording.mp4' -> 's3://videos.niteo.co/IRL4/IRL_talk_recording.mp4'  [part 1 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in   45s   111.64 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://videos.niteo.co/IRL4/IRL_talk_recording.mp4'  [part 2 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in   37s   136.04 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://videos.niteo.co/IRL4/IRL_talk_recording.mp4'  [part 3 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in   25s   204.51 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://videos.niteo.co/IRL4/IRL_talk_recording.mp4'  [part 4 of 211, 5MB] [1 of 1]
  524288 of 5242880    10% in    3s   155.49 kB/s^CERROR:
Upload of 'IRL_talk_recording.mp4' part 4 failed. Use
  bin/s3cmd abortmp s3://videos.niteo.co/IRL4/IRL_talk_recording.mp4 zOidfC...5RHKek_DS
to abort the upload, or
  bin/s3cmd --upload-id zOidfC...5RHKek_DS put ...
to continue the upload.
See ya!

~$ s3cmd --upload-id zOidfC...5RHKek_DS  put --multipart-chunk-size-mb=5 IRL_talk_recording.mp4 s3://videos.niteo.co/multi/
WARNING: MultiPart: size and md5sum match for s3://videos.niteo.co/IRL4/IRL_talk_recording.mp4 part 1, skipping.
WARNING: MultiPart: size and md5sum match for s3://videos.niteo.co/IRL4/IRL_talk_recording.mp4 part 2, skipping.
WARNING: MultiPart: size and md5sum match for s3://videos.niteo.co/IRL4/IRL_talk_recording.mp4 part 3, skipping.
upload: 'IRL_talk_recording.mp4' -> 's3://videos.niteo.co/IRL4/IRL_talk_recording.mp4'  [part 4 of 211, 5MB] [1 of 1]
  5242880 of 5242880   100% in  230s    22.17 kB/s  done
upload: 'IRL_talk_recording.mp4' -> 's3://videos.niteo.co/IRL4/IRL_talk_recording.mp4'  [part 5 of 211, 5MB] [1 of 1]
  2097152 of 5242880    40% in   60s    33.59 kB/s

Success! Glad I finally found this. Today is not the first time I got failed S3 uploads on flaky internet connections. Now I know how to resume them!