S3 bucket upload task exceeds the 5GB Limit
Posted: Thu Mar 07, 2019 9:14 am
S3 bucket upload task exceeds the 5GB Limit
Possble Error in Log:
Your proposed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLarge; Request ID: XXX). Full stack trace written to '#####_error_1.log'
AWS size limit 5TB. https://aws.amazon.com/blogs/aws/amazon ... ize-limit/
Reason:
The S3 Task's Upload action does a direct PUT of the entire file in 1 request. The main purpose of the specific S3 Task is to work with an object's metadata. Even though GoAnywhere supports uploading files from the task (PUT), it doesn't use the multiple part upload, which is needed to support larger files.
Resolution:
GoAnywhere does support the multiple part upload to S3 Buckets if using the resource:s3:// syntax (File System - Copy). The project should use the Copy Task with the destination as the resource:s3:// syntax (or use the file chooser "..." > Resource Links > Amazon S3) This will take advantage of the multiple part upload and allow for files over 5GB to be uploaded to S3.
Possble Error in Log:
Your proposed upload exceeds the maximum allowed size (Service: Amazon S3; Status Code: 400; Error Code: EntityTooLarge; Request ID: XXX). Full stack trace written to '#####_error_1.log'
AWS size limit 5TB. https://aws.amazon.com/blogs/aws/amazon ... ize-limit/
Reason:
The S3 Task's Upload action does a direct PUT of the entire file in 1 request. The main purpose of the specific S3 Task is to work with an object's metadata. Even though GoAnywhere supports uploading files from the task (PUT), it doesn't use the multiple part upload, which is needed to support larger files.
Resolution:
GoAnywhere does support the multiple part upload to S3 Buckets if using the resource:s3:// syntax (File System - Copy). The project should use the Copy Task with the destination as the resource:s3:// syntax (or use the file chooser "..." > Resource Links > Amazon S3) This will take advantage of the multiple part upload and allow for files over 5GB to be uploaded to S3.